Extended Legacy Format (ELF):
Serialisation Format

FHISO’s Extended Legacy Format (or ELF) is a hierarchical serialisation format and genealogical data model that is fully compatible with GEDCOM, but with the addition of a structured extensibility mechanism. It also clarifies some ambiguities that were present in GEDCOM and documents best current practice.

The GEDCOM file format developed by The Church of Jesus Christ of Latter-day Saints is the de facto standard for the exchange of genealogical data between applications and data providers. Its most recent version is GEDCOM 5.5.1 which was produced in 1999, but despite many technological advances since then, GEDCOM has remained unchanged.

FHISO are undertaking a program of work to produce a modernised yet backward-compatible reformulation of GEDCOM under the name ELF, the new name having been chosen to avoid confusion with any other updates or extensions to GEDCOM, or any future use of the name by The Church of Jesus Christ of Latter-day Saints. This document is one of five that form the initial suite of ELF standards, known collectively as ELF 1.0.0:

• ELF: Primer. This is not a formal standard, but is being released alongside the ELF standards to provide a broad overview of ELF written in a less formal style. It gives particular emphasis to how ELF differs from GEDCOM.

• ELF: Serialisation Format. This standard defines a general-purpose serialisation format based on the GEDCOM data format which encodes a dataset as a hierarchical series of lines, and provides low-level facilities such as escaping.

• ELF: Schemas. This standard defines flexible extensibility and validation mechanisms on top of the serialisation layer. Although it is an optional component of ELF 1.0.0, future ELF extensions to ELF will be defined using ELF schemas.

• ELF: Date, Age and Time Microformats. This standard defines microformats for representing dates, ages and times in arbitrary calendars, together with how they are applied to the Gregorian, Julian, French Republican and Hebrew calendars.

• ELF: Data Model. This standard defines a data model based on the lineage-linked GEDCOM form, reformulated to be usable with the ELF serialisation model and schemas. It is not a major update to the GEDCOM data model, but rather a basis for future extension and revision.

Conventions used

Where this standard gives a specific technical meaning to a word or phrase, that word or phrase is formatted in bold text in its initial definition, and in italics when used elsewhere. The key words must, must not, required, shall, shall not, should, should not, recommended, not recommended, may and optional in this standard are to be interpreted as described in [RFC 2119].

An application is conformant with this standard if and only if it obeys all the requirements and prohibitions contained in this document, as indicated by use of the words must, must not, required, shall and shall not, and the relevant parts of its normative references. Standards referencing this standard must not loosen any of the requirements and prohibitions made by this standard, nor place additional requirements or prohibitions on the constructs defined herein.

This standard depends on FHISO’s Basic Concepts for Genealogical Standards standard. To be conformant with this standard, an application must also be conformant with the referenced parts of [Basic Concepts]. Concepts defined in that standard are used here without further definition.

Certain facilities in this standard are described as deprecated, which is a warning that they are likely to be removed from a future version of this standard. This has no bearing on whether a conformant application must implement the facility: they may be required, recommended or optional as described in this standard.

Indented text in grey or coloured boxes does not form a normative part of this standard, and is labelled as either an example or a note.

The grammar given here uses the form of EBNF notation defined in §6 of [XML], except that no significance is attached to the capitalisation of grammar symbols. Conforming applications must not generate data not conforming to the syntax given here, but non-conforming syntax may be accepted and processed by a conforming application in an implementation-defined manner, providing a warning is issued to the user, except where this standard says otherwise.

The grammar productions in this standard uses the S and Char productions defined in §2 of [Basic Concepts] to match any non-empty sequence of whitespace characters or any valid character, respectively.

Overview

The ELF serialisation format is a structured, line-based text format for encoding data in a hierarchical manner that is both machine-readable and human-readable.

At a logical level, an ELF document is built from structures, the name ELF gives to the basic hierarchical data structures used to represent data. ELF uses two types of structure: tagged structures and typed structures. The serialisation layer described in this standard only deals with tagged structures, and the word structure is frequently used in this document to refer to what is properly a tagged structure.

A tagged structure consists of:

• an optional cross-reference identifier used to identify the structure within the document;
• a tag, which is a string encoding the meaning of the structure;
• an optional payload, which can be considered the value of the structure; and
• a sequence of zero or more child structures known as its substructures. These can nest arbitrarily deep in a hierarchical manner.

The tag describes how the structure is to be interpreted, and structures are commonly referred to by their tag in this standard.

The payload is either a language-tagged string or a pointer to another structure. A payload which is a language-tagged string is referred to as a string payload.

When the payload of a structure is a pointer, this represents a link between two structures, with the pointer in one structure referencing the cross-reference identifier in a second structure.

A top-level structure, meaning a structure which is not a substructure of any other structure, is called a record. An ELF document or dataset can have arbitrarily many records.

At a lexical level, a structure is encoded as sequence of lines, each terminated with a line break. The first line encodes the cross-reference identifier, tag and payload of the structure, while any substructures are encoded in order on subsequent lines. Each line consists of the following components, in order, separated by whitespace:

• a level, which is an integer that records how deeply the current structure is nested;
• the optional cross-reference identifier of the structure being encoded by the line;
• the tag of the structure being encoded; and
• the optional payload of the structure.

0 HEAD
1 CHAR UTF-8
1 GEDC
2 VERS 5.5.1
2 FORM LINEAGE-LINKED
1 ELF 1.0.0
0 INDI
1 NAME Charlemagne
0 TRLR

ELF applications

A conformant application which parses the ELF serialisation format is called an ELF parser. A conformant application which outputs data in the ELF serialisation format is called an ELF writer.

The input to an ELF parser and output of an ELF writer is an octet stream, which is a sequence of 8-bit bytes or octets each with a value between 0 and 255.

This standard defines how an octet stream is parsed into a dataset, and how a dataset is serialised into an octet stream. Overviews of these processes can be found in §2.2 and §2.3, respectively. An octet stream which this standard requires an ELF parser to be able to read is called a conformant source.

An octet stream which is not a conformant source is called a non-conformant source. If the input to an ELF parser is not a conformant source, unless this standard says otherwise, the application must either terminate processing that octet stream or present a warning or error message to the user. If it continues processing, it does so in an implementation-defined manner.

This standard also recognises a class of application which reads data in the ELF serialisation format, applies a small number of changes to that data, and immediately produces output in the ELF serialisation format which is identical to the input, octet for octet, other than where the requested changes have been made. Such an application is called an ELF editor.

ELF editors are not required to conform to the full requirements of an ELF parser or ELF writer. The only requirement this standard places on ELF editors is that, when acting on a conformant source, they must either generate output which is a conformant source, or present a warning or error message to the user, or terminate.

This standard has an optional dependency on the [ELF Schemas] standard, which provides additional functionality for validating ELF documents and extending the ELF data model. An application which conforms to the [ELF Schemas] standard is described as schema-aware; other applications are described as non-schema-aware.

Parsing

The parsing process can be summarised as follows:

1. An octet stream is converted to a sequence of line strings by:

   1. determining its character encoding by

      1. identifying the detected character encoding per §3.1, and
      2. using that detected character encoding to look for a specified character encoding in the serialisation metadata per §3.2;
   2. converting octets to characters using that character encoding; and

   3. splitting on line breaks per §3.4.

2. Line strings are converted into records by:

   1. parsing line strings into lines per §4.1;
   2. assembling lines into records, each of which are hierarchies of tagged structures, as described in §4.2.1.

3. The header record is parsed for serialisation metadata per §5.2.

4. A second pass is made recursively over each record, processing it per §4.2.2:

   1. if the parser is schema-aware, converting tagged structures into typed structures, as described in [ELF Schemas]; and

   2. each string payload is unescaped by:

      1. identifying all escaped at signs and escape sequences per §6.5.1;
      2. verifying that each escape sequence is a permitted escape per §6.5.2;
      3. replacing each escaped at sign with a single “at” sign;
      4. replacing each Unicode escape with the character it encodes per §6.3; and
      5. merging continuation lines per §6.5.3.

Serialisation

The semantics of serialisation are defined by the following procedural outline.

2. The tagged structures are ordered and additional tagged structures created to represent serialisation metadata.

   This step cannot happen before tagging because tagging may generate serialisation metadata that needs to be included in the tagged structures.

3. Payloads are converted to create xref structures by simultaneously

   • assigning xref_ids and replacing pointer-valued payloads with string-valued xrefs
   • escaping @ characters
   • preserving valid escapes
   • escaping unrepresentable characters

   Semantically, these actions must happen concurrently because none of them should be applied to the others’ results.

   This step cannot happen before tagging because tags are needed to determine the set of valid escapes. This step cannot happen before adding serialisation metadata because it is applied to the serialisation metadata as well.

4. The dataset is converted to a sequence of lines by

   • assigning levels
   • splitting payloads, if needed, using CONT and CONC
   • ordering substructures in a preorder traversal of the tagged structures

   This step cannot happen before payload conversion because valid split points are dependant on proper escaping. This step must happen before encoding as octets because valid split points are determined by character, not octet.

5. The sequence of lines is converted to an octet stream by

   • concatenating the lines with line-break terminators
   • converting strings to octets using the character encoding

Glossary

Dataset

Metadata and a document.

Document

An unordered set of structures.

Metadata

A collections of structures intended to describe information about the dataset as a whole.

The relative order of structures with the same structure type identifier SHALL be preserved within this collection; the relative order of structures with distinct structure type identifiers is not defined by this specification.

ELF Schema

Information needed to correctly parse tagged structures into structures: a mapping between structure type identifiers and tags and metadata relating to valid escapes and prefixes.

Serialisation Metadata

Tagged structures inserted during serialisation and removed (with all its substructures) during parsing. They are used to serialise the character encoding and ELF schema as well as to separate the metadata and the document.

Structure

• A structure type identifier, which is a term.
• Optionally, a payload which is one of
  • A pointer to another structure, which must be a record within the same dataset.
  • A string or subtype thereof.
• One superstructure, which is one of
  • Another structure; superstructure links MUST be acyclic.
  • The document.
  • The metadata.

• A collection of any number of substructures, which are structures.

  The relative order of structures with the same structure type identifier SHALL be preserved within this collection; the relative order of structures with distinct structure type identifiers is not defined by this specification.

Parsing and serialising line strings

In order to parse an ELF document, an ELF parser shall first convert the octet stream into a sequence of line strings, which are strings containing the unparsed lexical representations of lines.

The way in which octets are mapped to characters is called the character encoding of the document. ELF supports several different character encodings. Determining which is used is a two-stage process, with the first stage being to determine the detected character encoding of the octet stream per §3.1. Frequently there will be no detected character encoding.

Next, the initial portion of the octet stream is converted to characters using the detected character encoding, failing which in an ASCII-compatible manner. This character sequence is then scanned for a CHAR line whose payload identifies the specified character encoding. This process is described in §3.2. If there is a specified character encoding, it is used as the character encoding for the ELF document; otherwise the detected character encoding is used, failing which the default is the ANSEL character encoding. Considerations for reading specific character encodings can be found in §3.3.

Once the character encoding is determined, the octet stream can be converted into a sequence of characters which are assembled into line strings as described in §3.4. The process of serialising a line string back into an octet stream is far simpler as the intended character encoding is already known; this process is described in §3.5.

Detecting a character encoding

If the octet stream begins with a byte-order mark (U+FEFF) encoded in UTF-8, the detected character encoding shall be UTF-8; or if the application supports the optional UTF-16 encoding and the octet steam begins with a byte-order mark encoded in UTF-16 of either endianness, the detected character encoding shall be UTF-16 of the appropriate endianness. The byte-order mark shall be removed from the octet stream before further processing.

Otherwise, if the application supports the optional UTF-16 encoding and the octet stream begins with any ASCII character (U+0001 to U+007F) encoded in UTF-16 of either endianness, this encoding shall be the detected character encoding.

Otherwise, applications may try to detect other encodings by examining the octet stream in an implementation-defined manner, but this is not recommended.

Otherwise, there is no detected character encoding.

Specified character encodings

To determine the specified character encoding, the initial portion of the octet stream shall temporarily be converted to characters using the detected character encoding.

If there is no detected character encoding, the application shall convert each octet to the character whose code point is the value of octet. An application shall issue an error and stop processing the octet stream if the null octet 00 is encountered. Restricted characters, as defined in §2.3 of [Basic Concepts], must be accepted without error while determining the specified character encoding.

Characters from the initial portion of the octet stream are parsed into lines strings as described in §3.4. Each line string is whitespace normalised as described in §2.1 of [Basic Concepts], and all lowercase ASCII characters (U+0061 to U+007A) converted to the corresponding uppercase characters (U+0041 to U+005A).

Once normalised in this manner, the first line string of the file must be exactly “0 HEAD”; otherwise the application must issue an error and cease parse the octet stream as ELF. If the application encounters a subsequent normalised line string beginning with a 0 digit (U+0030) followed by a space character (U+0020), the application shall stop scanning for a specified character encoding.

If the application encounters a line string beginning with “1 CHAR” followed by a space character (U+0020) while scanning for the specified character encoding, then the remainder of the line string shall be used to determine the specified character encoding.

If the remainder of the line string is exactly “ASCII”, “ANSEL” or “UTF-8”, then the specified character encoding shall be ASCII, ANSEL or UTF-8, respectively.

0 HEAD
1 CHAR UTF-8

Otherwise, if the remainder of the line string is exactly “UNICODE” and the detected character encoding is UTF-16 in either endianness, the specified character encoding shall be the UTF-16 in that endianness.

Otherwise, the application may determine the specified character encoding from the remainder of the line string and the detected character encoding in an implementation-defined way. The application may read one further line string, and if it begins with “2 VERS” followed by a space character (U+0020), the application may also use the remainder of that line string in determining the specified character encoding.

0 HEAD
1 CHAR ANSI
2 VERS 1250

Otherwise, there is no specified character encoding.

If there is a specified character encoding, it shall be used as the character encoding of the octet stream. Otherwise, if there is a detected character encoding, it shall be used as the character encoding of the octet stream. Otherwise, the character encoding shall default to be UTF-8.

If the character encoding is one which the application does not support, the application shall issue an error and stop reading the file.

Character encodings

ELF parsers are required to support reading the ASCII, ANSEL and UTF-8 character encodings. ELF writers are only required to support the UTF-8 character encoding. Support for the UTF-16 character encoding is optional, and applications may support it in either its big or little endian forms, both, or neither. The ASCII, ANSEL and UTF-16 character encodings are all deprecated.

The UTF-8 and UTF-16 character encodings are the Unicode encoding forms defined in §9.2 of [ISO 10646], and the specifics of the big and little endian forms of UTF-16 are defined in §9.3 of [ISO 10646].

The character encoding referred to as ASCII in this standard is the US version of ASCII which, for the purpose of this standard, is defined as the subset of UTF-8 which uses only Unicode characters U+0001 to U+007F.

ANSEL refers to the Extended Latin Alphabet Coded Character Set for Bibliographic Use defined in [ANSEL]. If an ELF file is determined to use the ANSEL character encoding it must be converted into a sequence of Unicode characters before it can be processed further. This is discussed in §3.3.1.

If other character encodings are supported, they too must be converted into a sequence of Unicode characters for further processing.

Converting ANSEL to Unicode

Line strings

Before characters from the octet stream can be parsed into lines, they must be assembled into line strings. This is done by appending characters to the line string until a line break is encountered, at which point the character or characters forming the line break are discarded and a new line string is begun.

ELF parsers must be able to handle arbitrarily long line strings, subject to limits of available system resources.

Any leading whitespace shall be removed from the line string, but trailing whitespace must not also be removed except in the case that the line string is entirely whitespace. If this results in a line string which is an empty string, the empty line string is discarded.

Serialising line strings

Line strings are serialised by concatenating them together to form a single string, inserting a line break between each line string and after the last one. All the inserted line breaks must have identical lexical forms.

Finally, the resulting string is encoded into an octet stream using the character encoding that was documented in the serialisation metadata tagged structure with tag “CHAR” (see §8.1). ELF writers are only required to support the UTF-8 character encoding, and this should be the default in applications supporting additional character encodings.

If the character encoding is one which allows a byte-order mark (U+FEFF) to be encoded, an ELF writer may prepend one the octet stream. This is recommended when serialising to UTF-16, but is not recommended when serialising to UTF-8.

Parsing and serialising structures

Parsing lines

For a line string to be parsed into a line, it must match the following Line production:

Line        ::=  Number S (XRefLabel S)? Tag (PayloadSep Payload)?
PayloadSep  ::=  #x20 | #x9

0 @I1@ INDI
1 NAME Cleopatra
1 FAMC @F2@

Malformed lines are lines or line strings which contain certain particular types of syntactic error. Input containing a malformed line is a non-conformant source. If an ELF parser encounters a malformed line, it shall terminate processing the input file.

Any line string which does not match the Line production is a malformed line.

Levels

The Number production encodes the level of the line, which is a non-negative decimal integer that records how many levels of substructures deep the current structure is nested.

Number  ::=  "0" | [1-9] [0-9]*

The previous level of a line is defined as the level of the closest preceding line. The first line in the input stream has no previous level.

0 INDI
1 NOTE The 16th President of the United States.
2 CONT Assassinated by John Wilkes Booth.
0 TRLR

Any line that has a level more than one greater than its previous level is a malformed line. This does not apply to the first line in the input stream which is never a malformed line.

0 @I1@ INDI
2 PLAC Москва
3 ROMN Moscow
1 NAME Иван Васильевич
0 TRLR

Cross-reference identifiers

The XRefLabel production encodes the cross-reference identifier of the line, which is used when referencing one structure from another using a pointer, and may be omitted when there is no need to refer to the structure. It is encoded with an “at” signs (@; U+0040) before and after it, which are not themselves part of cross-reference identifer.

XRefLabel  ::=  "@" XRefID "@"
XRefID     ::=  IDChar+
IDChar     ::=  [A-Za-z0-9] | [?$&'*+,;=._~-]
                  | [#xA0-#xD7FF] | [#xF900-#xFFEF] | [#x10000-#xEFFFF]

0 @I1@ INDI

For maximum compatibility, ELF writers should prefer cross-reference identifiers which only use ASCII characters, and should make the first character of a cross-reference identifier a letter (U+0041 to U+005A or U+0061 to U+007A), decimal digit (U+0030 to U+0039) or underscore (U+005F).

Tags

The Tag production encodes the tag of the line which is a required string that denotes the meaning of the data encoded on the line.

Tag  ::=  [0-9a-zA-Z_]+

The ELF suite of standards defines a selection of tags for representing genealogical data.

Third parties may define additional tags for use in ELF documents in two ways. The first way, which is deprecated, is to use a legacy extension tag. These are tags beginning with an underscore (_, U+005F). No legacy extension tags are defined in the ELF standards, and third parties can use them arbitrarily.

1 _UID 40ea7ad8-a5ba-4a7a-bb89-615cc2bf6639

The second and preferred means of adding third-party tags is to define them in an ELF schema and reference that schema using a schema reference.

The HEAD, TRLR, CONC, CONT, PLANG and DTYPE tags are reserved in all contexts for recording header records, trailer records, continuation lines, payload languages and payload datatypes and must not be used in any other way.

A tag should be no more than 15 characters in length.

Payloads

The payload of a line is an optional value associated with the line, which is encoded by the Payload production. If present, it shall be either a string or a pointer, which are encoded by the PayloadString and Pointer productions, respectively. The String production is given in §2 of [Basic Concepts] as a sequence of zero or more characters.

Payload        ::=  S? Pointer S? | PayloadString
PayloadString  ::=  String - ( S? Pointer S? )

Applications must treat a line with an omitted payload identically to a line with a payload consisting of an empty string.

PayloadString ::= PayloadItem*
PayloadItem   ::= PayloadChar | EscapedAt | EscapeSeq
PayloadChar   ::= [^#x40#xA#xD]
EscapedAt     ::= "@@"
EscapeSeq     ::= "@#" [A-Z] PayloadChar* "@"

A pointer is a payload which represents a link to another structure. It is encoded using the following Pointer production.

Pointer  ::=  "@" [^#x23#x40#xA#xD] [^#x40#xA#xD]* "@"

GEDCOMPointer  ::=  "@" (IDChar+ ":")? XRefID ("!" IDChar+)? "@"

Parsing lines into structures

Once line strings have been parsed into lines, the sequence of lines is converted into a sequence of records.

This process starts by parsing the first line of the input as the first line of a tagged structure using the procedure given §4.2.1. If that record has substructures then additional lines will be read while parsing it. This structure is the first record in the dataset, and shall be the header record.

Once the header record has been read, it shall be parsed according to §5.2 to extract the serialisation metadata, which affects the subsequent parsing of the file.

If further lines remain after the header record has been fully parsed, then the first of the remaining lines is parsed as first line of the next record in the dataset, again using the procedure given in §4.2.1. This process is repeated until no further lines remain, at which point the dataset has the been fully read.

If the last record has a tag of TRLR, and no cross-reference identifier, payload or substructures, it is discarded. Such a record is called a trailer record. If the last record is not a trailer record, it is a malformed structure as defined in §4.2.3.

Once each record has been assembled, an ELF parser shall make a second pass over the record processing it as described in §4.2.2. This does not apply to the discarded trailer record.

First pass: assembling

The conversion of lines into structures is defined recursively. To read a structure, the parser starts by reading its first line, and creates a tagged structure whose components are as follows:

• the cross-reference identifier of the first line;
• the tag of the first line;
• the payload of the first line, provisionally tagged with the “und” language tag if the payload of the first line is a string rather than a pointer; and
• an empty sequence of substructures.

The level of the first line of the structure is referred to in this section as the current level.

The parser then repeatedly inspects the next line to determine whether it represents the start of a substructure of the structure being read. If the next line has a level less than or equal to the current level, there are no further substructures and the application has finished reading the structure.

1 DEAT Y
0 TRLR

Otherwise, the application shall recursively parse the next line as the first line of a new structure and append it to the list of substructures being read. Parsing continues by inspecting the following line to see if it is the start of another substructure, as described above.

0 @I1@ INDI
1 NAME Elizabeth
1 BIRT
2 DATE 21 APR 1926
0 TRLR

Second pass: processing

Once each of record has been assembled, an ELF parser shall make a second pass over the record, processing it and its substructures recursively. Each step of the recursion proceeds as follows.

First, if the structure has a tag of CONC, CONT or TRLR, or if the tag is HEAD and the structure is not the first record of the input, it is a malformed structure.

Next, if the ELF parser is schema-aware, the tagged structure shall be converted into a typed structure as described in [ELF Schemas].

Next, if the payload of structure is a string payload, it is unescaped as described in §6.5.

Finally, each substructure of the structure is processed recursively, in order, as described in this section.

Errors in structures

This standard defines two classes of error that can arise when processing a structure.

A malformed structure is a structure with a sufficiently serious error that an ELF parser must detect the error and must terminate processing the input file upon encountering one.

A non-conformant structure is a structure with a less serious error. Input containing either a malformed structure or a non-conformant structure is a non-conformant source.

Serialising structures

Each xref structure is encoded as a sequence of one or more lines.

These are of three kinds, in order:

1. The first line of the xref structure
2. Zero or more additional lines of the xref structure
3. The lines that encode each of the xref structure’s substructures (if any)

The level of each line is a non-negative integer. The level of a first line is 0 if the xref structure is a record or the serialisation metadata tagged structures with tag “HEAD” and “TRLR”; otherwise it is one greater than the level of the first line of its superstructure. The level of an additional line is one greater than the level of its xref structure’s first line.

Each first line has the same xref_id (if any) and tag as its corresponding xref line. Each additional line has no xref_id and either “CONT” or “CONC” as its tag.

The payload of the xref structure is the concatenation of the payloads of the first line and all additional lines, with a line break inserted before the payload of each additional line with tag “CONT”. Because the payload of a line must not contain a line-break, there must be exactly one “CONT”-tagged additional line per line-break in the xref structure’s payload. The number of “CONC”-tagged additional lines may be picked arbitrarily, subject to the following:

• Each line should be no more than 255 octets after a line break has been added and the result encoded in the target character encoding. This recommended limit is increased to 510 octets if the target character encoding is UTF-16.

• The payload of a line preceding a “CONC”-tagged line should not have an empty payload.
• The payload of a line preceding a “CONC”-tagged line must NOT end with a whitespace.
• A “CONC”-tagged line’ payload should not begin with whitespace.

2 NOTE This is a test
3 CONT with one line break

2 NOTE This i
3 CONC s a test
3 CONT with on
3 CONC e line break

• Each line’s payload must contain an even number of U+0040 (@). However, during parsing, this constraint SHALL NOT be enforced in any way.

1 EMAIL name@example.com
2 DATE @#DGREG
3 CONC ORIAN@ 2 JAN 2019

1 EMAIL name@example.com
2 DATE @#DGREGORIAN@ 2 JAN 2019

1 EMAIL name@@example.com
2 DATE @#DGREGORIAN@ 2 JAN 2019

Serialising lines

Each line shall be converted to a line string by concatenating together the level, cross-reference identifier, tag and payload as described by the Line production given in §4.1. The application must serialise all line strings with a single space character (U+0020) for each S or PayloadSep production in the Line production, and must not put additional whitespace before or after payloads which are pointers.

1 FAMC  @F9@

Header metadata

The header record is the first record in an ELF document. It shall have a HEAD tag, no payload and no cross-reference identifier. The substructures of the header record are called metadata structures, and contain information about the dataset as a whole.

Certain metadata structures, which are referred to as serialisation metadata structures, are processed by the ELF parser during parsing and then removed from the dataset. Each serialisation metadata structure encodes one piece of serialisation metadata, as determined by the tag of the serialisation metadata structure. The serialisation metadata affects how the ELF parser processes the file.

This standard defines five types of serialisation metadata, as given in the following table.

0 HEAD
1 ELF 1@#U2E@0

0 HEAD
1 NOTE Ceci est une note longue @#UC0@ pro
2 CONC pos de ce document
2 PLANG fr
0 TRLR

Version numbers

The payload of the ELF serialisation metadata structure, and the payload of the VERS substructure of the GEDC serialisation metadata structure both contain a version number, which is a string used to record the version of a standard that matches the following Version production:

Version  ::=  Integer "." Integer ( "." Integer )?
Integer  ::=  [0-9]+

The three components represented by the Integer production are decimal integers, and may include leading zeros which are ignored. These components are called the major version, minor version and revision number, respectively. If the revision number is omitted, a value of 0 is assumed.

1 ELF 1.0
1 ELF 1.0.0
1 ELF 1.000

ELF serialisation version

The ELF serialisation version is a version number located in the payload of the ELF serialisation metadata structure, and indicates the version of the ELF Serialisation standard with which the document complies.

The version number of this version of the standard is 1.0.0. An ELF writer producing output according to this standard must include this ELF serialisation version in the output if the generated file contains any Unicode escapes, schema references, payload languages or payload datatypes.

If an ELF parser is reading a document with an ELF serialisation version which differs from the version number of this standard only by the revision number, the ELF parser must parse the input according to this standard.

If an ELF parser encounters an ELF serialisation version which has a different minor version to this standard, but the same major version, it should parse the input according to this standard, but should issue a warning to the user that the document is in an unknown version of ELF.

If an ELF parser encounters an ELF serialisation version with a different major version, the document is a non-conformant source.

Legacy GEDCOM version

The legacy GEDCOM version is a version number located in the payload of the VERS substructure of the GEDC serialisation metadata structure, and indicates the version of GEDCOM which the document is compatible with.

This standard, when used together with the [ELF Data Model], is compatible with both GEDCOM 5.5 and GEDCOM 5.5.1. An ELF writer producing output according to this standard must include a legacy GEDCOM version of either 5.5 or 5.5.1 in the output if it omitted the ELF serialisation version or if it included no schema references in the output, and should do so otherwise if the document conforms to the [ELF Data Model].

If an ELF parser encounters a legacy GEDCOM version other than 5.5 or 5.5.1, the document is a non-conformant source.

0 HEAD
1 GEDC
2 VERS 5.3

Parsing serialisation metadata

Once a header record has been assembled as described in §4.2.1, the ELF parser shall iterate over its substructures looking for structures with a tag of CHAR, ELF, GED, PLANG or SCHMA. These substructures are identified as serialisation metadata structures and each is processed as specified in this section.

Any serialisation metadata structure, or any structure nested within a serialisation metadata structure regardless of the depth of the nesting, is a non-conformant structure if it has a cross-reference identifier, or if it has a tag of HEAD, TRLR, CONC or CONT, or if it has a payload which is a pointer.

0 HEAD
1 SCHMA https://example.com/this/is/a/very/long/IRI
2 CONC /which/has/been/continued/on/to/two/lines
0 TRLR

If a header record contains two or more serialisation metadata structures with the same tag, and that tag is not SCHMA, the second and subsequent serialisation metadata structures are non-conformant structures.

0 HEAD
1 PLANG nds
1 PLANG de

If the serialisation metadata structure has a tag of CHAR, it is deleted from the header record with no further processing.

If the serialisation metadata structure has a tag of ELF, and its payload is not a valid version number, it is a non-conformant structure. Otherwise, the version number in its payload is interpreted as the ELF serialisation version as described in §5.1.1, and the structure is deleted from the header record.

0 HEAD
1 ELF 1.0

If the serialisation metadata structure has a tag of GEDC, it is used to determine the legacy GEDCOM version as follows. The serialisation metadata structure is a non-conformant structure if it has a payload, or if it does not have exactly one substructure with a VERS tag and exactly one substructure with a FORM tag, or if the payload of the VERS substructure is not a valid version number, or if the payload of the FORM substructure is not the string “LINEAGE-LINKED”. Otherwise, the version number in the payload of the VERS substructure is interpreted as the legacy GEDCOM version as described in §5.1.2, and the whole serialisation metadata structure is deleted from the header record.

0 HEAD
1 GEDC
2 VERS 5.5
2 FORM LINEAGE-LINKED

0 HEAD
1 GEDC
2 VERS 5.5.1 EL

Escaping

Once structures have been assembled from the lines forming them, and converted to a typed structures if the application is schema-aware, any string payloads need to be unescaped.

ELF uses the “at” sign (@; U+0040) in the representation of pointers, as well as in escape sequences which are used to encode a special processing instructions in a string payload. Other uses of the “at” sign in payloads which are strings should be escaped, and must be when not escaping it would result in an ambiguity.

ELF provides two escape mechanisms which can escape an “at” sign in a payload. The recommended mechanism is to use an escaped at sign, defined in §6.1. The alternative is to use a Unicode escape, which is a more general escape mechanism defined in §6.3 that allows arbitrary Unicode characters to be encoded. Unicode escapes are an example of an escape sequence, which is a general facility for embedding special processing instructions in a string payload. Escape sequences are defined in §6.2.

Escaped at signs

An escaped at sign is a string matching the EscapedAt production below, and is used to represent a single “at” sign in a string payload.

EscapedAt  ::=  "@@"

1 EMAIL name@@example.com

Escape sequences

An escape sequence is a string that can be used in a string payload to denote some form of special processing instruction.

An escape sequence shall match the following EscapeSeq production.

EscapeSeq    ::=  "@#" EscapeType EscapeValue "@"
EscapeType   ::=  [A-Z]
EscapeValue  ::=  [^#x40#xA#xD]*

2 DATE @#DFRENCH R@ 6 COMP 11

1 NAME Jo@#UE3@o

The escape type of an escape sequence is the single character matched by EscapeType production. It defines how the escape sequence is to be interpreted. This standard defines one escape type: the character U is used to represent Unicode escapes, as defined in §6.3.

This standard reserves all possible escape types for future FHISO use. Third parties must not use their own escape sequences, except as permitted by a FHISO standard.

0 SCHMA
1 ESC B https://example.com/binary-escape

The escape value of an escape sequence is the string matched by the EscapeValue production. The meaning of the escape value and any restrictions on its content or format depend on the particular escape type. The only general restriction placed on all escape values is that they must not contain the “at” sign (U+0040), line feed (U+000A), or carriage return (U+000D).

@#T<https://userinfo%{40}example.com/>@

Unicode escapes

A Unicode escape is a type of escape sequence that allows arbitrary Unicode characters to be encoded ELF files, regardless of the character encoding used for the file. ELF parsers are required to support Unicode escapes.

Unicode escapes use the U escape type and has an escape value which is a sequence of zero or more uppercase hexadecimal integers, separate by spaces. The hexadecimal integers are the code points of the characters encoded by the Unicode escape. Its escape value shall matches the following UnicodeEsc production.

UnicodeEsc  ::=  S? ( HexNumber (S HexNumber)* S? )?
HexNumber   ::=  [0-9A-F]+

1 NAME Jo@#UE3@o

1 NAME Joa@#U303@o

1 NAME عزيز
1 NAME @#U639@@#U632@@#U64A@@#U632@
1 NAME @#U 639 632 64A 632@

ELF writers must use a Unicode escape to encode characters that cannot be encoded in the target character encoding, but should not use them otherwise without a specific need, and should prefer an escaped at sign to a Unicode escape when escaping an “at” sign (U+0040).

Line continuation

ELF allows the string payload of a structure to be split across two or more consecutive lines. When this is done, the first line which contains the start of the string payload is called the continued line and the subsequent line or lines which contain the remainder of the string payload are called continuation lines. Any line with a tag of CONT or CONC is a continuation line.

CONT continuation lines are used when the value encoded in a string payload needs to contain line breaks. The part of the string payload following each line break is placed on a continuation line using the CONT tag, and the line break itself is removed from encoded version of the payload.

4 TEXT Pray for the soule of Edward Cowrtney esquyer secunde son
5 CONT of sr Willm Cowrtney knyght of Povderam, which dyed the 
5 CONT firrst day of mch Ano dom mvcix on whos soule ihu have mci

CONC continuation lines are used when it is desirable to split a string payload which does not contain a convenient line break across several lines. The payload is split at an arbitrary place which should be between two characters that are not whitespace.

1 NOTE Prof. D. H. Kelley speculates that the mother of King Ecg
2 CONC berht of Wessex was a daughter of Æthelbeorht II of Kent.

Applications must not assign significance to where CONC continuation lines are inserted nor to how many are present in the serialisation of a string payload.

1 NOTE This is a long line which has been
2 CONSP split using the new mechanism.

Unescaping string payloads

In order to unescape a string payload of a structure, an ELF parser shall first identify all escaped at signs and escape sequences in the string payload per §6.5.1, and verify that each identified escape sequence is a permitted escape for the structure in whose payload it was found, as described in §6.5.2.

Next, each identified escaped at sign is replaced with a single “at” sign, and each identified Unicode escape is replaced with the character it encodes. Escape sequences other than Unicode escapes are left unaltered.

0 NOTE @@#U40@@
0 NOTE @#U40@@#U40@

Finally, any substructures corresponding to continuation lines are identified and their payloads merged into the payload of their parent structure, as described in §6.5.3.

0 NOTE @
1 CONC #U21@

Identifying escapes

To identify all the escaped at signs and escape sequences in a string payload, an ELF parser scans the string from beginning to end looking for “at” signs (U+0040), and then inspects the next character, if there is one, to determine how the “at” sign is to be interpreted.

If the following character is another “at” sign, then an ELF parser shall identify the two “at” signs as an escaped at sign, and then resume scan for “at” signs from the character following the second “at” sign.

1 EMAIL name@@example.com

Otherwise, if the following character is the number sign (#; U+0023), then an ELF parser shall identify these two characters as the start of an escape sequence, terminating at the subsequent “at” sign. If there is no subsequent “at” sign, or if the string identified as an escape sequence does not match the EscapeSeq production, the structure containing this string payload is a non-conformant structure. If a syntactically correct escape sequence was identified, the ELF parser shall resume scanning for “at” signs from the character following the second “at” sign.

0 NOTE Lines containing only a @# are non-conformant.

0 NOTE Following a @# with a @ isn't necessarily conformant.

Otherwise, the “at” sign is treated as a regular character, and scanning for “at” signs continues from the next character. This facility for treating unescaped “at” signs as regular characters is deprecated.

1 EMAIL name@example.com

Permitted escapes

A permitted escape is an escape sequence with an escape type that is permitted to occur in a particular structure. If a string payload contains an escape sequence other than an permitted escape, the structure is a non-conformant structure.

If the application is schema-aware, permitted escapes are identified as described in the [ELF Schemas] standard. Otherwise, permitted escapes are identified as described in this section.

If the escape type is U, then the escape sequence is a permitted escape.

If the escape type is D, then the escape sequence is a permitted escape.

1 DEAT
2 DATE @#DJULIAN@ 30 JAN 1649
2 AGE @#DJULIAN@ 48y

An escape sequence with any other escape type is not a permitted escape.

Merging continuation lines

Substructures with a tag of CONC or CONT are called a continuation substructures. They correspond to continuation lines.

A continuation substructure is a malformed structure if it has a cross-reference identifier, or has a non-empty list of substructures, or is a substructure of a continuation substructure, or is preceded in the list of substructures by a structure other than a continuation substructure. Likewise, any record whose tag is CONT or CONC is a malformed structure.

0 NOTE Start of note
1 REFN 5bb43407-9f24-4b42-b00e-c32cc0f09d21
1 CONT End of note

A continuation substructure is a non-conformant structure if it has a payload which is a pointer.

0 @N1@ NOTE This can be found in:
1 CONT @F1@

If a structure has any continuation substructures, each is merged with the parent structure in the order they appear in the list of substructures, as follows.

If a CONC continuation substructure is encountered, an ELF parser shall first append a line break to the payload of the parent structure. The form of line break appended is implementation-defined, but all inserted line breaks must have identical lexical forms.

Then, regardless of the type of continuation substructure, the payload of the continuation substructure shall be appended to the payload of the parent structure, and the continuation substructure is removed from the parent’s list substructures.

0 NOTE This paragraph is sufficiently long that it has proved con
1 CONC venient to wrap it onto a second line.
1 CONT
1 CONT This is a short paragraph.
1 REFN 8e445bb6-cb27-4c12-8c74-e051395639c2

Encoding with @

Pointer conversion

If a tagged structure is pointed to by the pointer-valued payload of another tagged structure, the pointe-to tagged structure’s corresponding xref structure shall be given an xref_id, a string matching production XrefID.

XrefID  ::= "@" ID "@"
ID      ::= [0-9A-Z_a-z] [#x20-#x3F#x41-#x7E]*

It must not be the case that two different xref structures be given the same xref_id. Conformant implementations must not attach semantic importance to the contents of an xref_id.

It is recommended that an xref_id be no more than 22 characters (20 characters plus the leading and trailing U+0040)

Each record should be given an xref_id; each non-record structure should not; and each serialisation metadata tagged structure must not be given an xref_id.

The xref structure that corresponds to a tagged structure with a pointer-valued payload has, as its payload, an xref: a string identical to the xref_id of the xref structure corresponding to the pointed-to tagged structure.

When parsing, if xref payloads are encountered that do not correspond to exactly one xref structure’s xref_id, that payload shall be converted to to a pointer to a record with tag “UNDEF”, which shall not have a payload nor substructures. It is recommended that one such “UNDEF” tagged structure be inserted for each distinct xref.

Escape preservation and removal

If the escape type is U (U+0055), the escape is a unicode escape and its handling is discussed in §6.3; otherwise, it is handled according to this section.

Serialisation

If an escape is in the payload of an tagged structure whose tag is an escape preserving tag, and if the escape’s escape type* is in the tag’s set of preserved escape types, then the escape shall be preserved unmodified in the corresponding xref structure’s payload.

Otherwise, a modification of the escape shall be placed in the xref structure’s payload which is identical to the original escape except that each of the two @ shall each be replaced with a pair of consecutive U+0040 @.

Parsing

If an escape is in the payload of an xref structure whose tag is an escape preserving tag, and the escape’s escape type* is in the tag’s set of preserved escape types, the escape shall be preserved unmodified in the corresponding tagged structure’s payload.

Otherwise, the escape shall be omitted from the corresponding tagged structure’s payload.

Encoding @s

During serialisation, each U+0040 (@) that is not part of an escape shall be encoded as two consecutive U+0040 (@@).

Serialisation metadata

The tagged structures representing the dataset are ordered as follows:

1. A serialisation metadata tagged structure with tag “HEAD” and the following substructures:

   • A serialisation metadata tagged structure with tag “CHAR” and payload identifying the character encoding used; see §8.1 for details.

   • A serialisation metadata tagged structure with tag “SCHMA” and no payload, with substructures encoding the ELF Schema.

   • Each tagged structure with the superstructure type identifier elf:Metadata, in an order consistent with the partial order of structures present in the metadata.

2. Each tagged structure with the superstructure type identifier elf:Document, in arbitrary order.

3. A serialisation metadata tagged structure with tag “TRLR” and no payload or substructures.

Charcter encoding names

The character encoding shall be serialised in the “CHAR” tagged structure’s payload encoding name in the following table:

It is required that the encoding used should be able to represent all code points within the string; unicode escapes (see §6.3) allow this to be achieved for any supported encoding. It is recommended that UTF-8 be used for all datasets.

References

Normative references

[ANSEL]

NISO (National Information Standards Organization). ANSI/NISO Z39.47-1993. Extended Latin Alphabet Coded Character Set for Bibliographic Use. 1993. (See http://www.niso.org/apps/group_public/project/details.php?project_id=10.) Standard withdrawn, 2013.

[Basic Concepts]

FHISO (Family History Information Standards Organisation). Basic Concepts for Genealogical Standards. Public draft. (See https://fhiso.org/TR/basic-concepts.)

[ELF Schema]

FHISO (Family History Information Standards Organisation) Extended Legacy Format (ELF): Schema.

[ISO 10646]

ISO (International Organization for Standardization). ISO/IEC 10646:2014. Information technology — Universal Coded Character Set (UCS). 2014.

[RFC 2119]

IETF (Internet Engineering Task Force). RFC 2119: Key words for use in RFCs to Indicate Requirement Levels. Scott Bradner, 1997. (See http://tools.ietf.org/html/rfc2119.)

[XML]

W3C (World Wide Web Consortium). Extensible Markup Language (XML) 1.1, 2nd edition. Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, Eve Maler, François Yergeau, and John Cowan eds., 2006. W3C Recommendation. (See https://www.w3.org/TR/xml11/.)

Other references

[ASCII]

ANSI (American National Standards Institute). ANSI X3.4-1986. Coded Character Sets – 7-Bit American National Standard Code for Information Interchange (7-Bit ASCII). 1986.

[GEDCOM 5.5]

The Church of Jesus Christ of Latter-day Saints. The GEDCOM Standard, release 5.5. 1996.

[GEDCOM 5.5.1]

The Church of Jesus Christ of Latter-day Saints. The GEDCOM Standard, draft release 5.5.1. 2 Oct 1999.

[ELF Data Model]

FHISO (Family History Information Standards Organisation) Extended Legacy Format (ELF): Data Model.

[ELF Dates]

FHISO (Family History Information Standards Organisation) Extended Legacy Format (ELF): Date, Age and Time Microformats. Public draft. (See https://fhiso.org/TR/elf-dates.)

[RFC 4122]

IETF (Internet Engineering Task Force). RFC 4122: A Universally Unique IDentifier (UUID) URN Namespace. P. Leach, M. Mealling and R. Salz, 2005. (See http://tools.ietf.org/html/rfc4122.)

[Unicode]

The Unicode Consortium. The Unicode Standard – Core Specification, version 12.1.0. See https://www.unicode.org/versions/Unicode12.1.0/.

[XML Names]

W3 (World Wide Web Consortium). Namespaces in XML 1.1, 2nd edition. Tim Bray, Dave Hollander, Andrew Layman and Richard Tobin, eds., 2006. W3C Recommendation. See https://www.w3.org/TR/xml-names11/.

Copyright © 2017–19, Family History Information Standards Organisation, Inc. The text of this standard is available under the Creative Commons Attribution 4.0 International License.