Skip to main content

Corpus Metadata Model

There are two separate, but related schemas for the LAUDATIO metadata: the first concerns the TEI-based metadata that are used as input from data providers. The second are JSON-based metadata that result from the TEI metadata being ingested into Elasticsearch. This set of metadata provides the attributes that are directly displayed in LAUDATIO.

The following tabs thus describe both sets of metadata and provide their validation sources (.rng in case of TEI metadata and .json in case of JSON). A separate sub-page contains some TEI file examples.

Preliminaries​

To better differentiate TEI tags, tag attributes, and attribute values, we display them like this:

  • tag
  • attribute
  • "value"

By default, all tags and attributes are required unless otherwise noted. We distinguish the following special cases:

  • ‼️: optional according to RNG schema, but required (or at least highly recommended) for LAUDATIO data ingestion
  • 0️⃣: optional and can be dropped
  • 🔂: its presence is dependent on another element, even if they are in theory independently optional (e.g. if forename is given, surname should also be given)

Basic structure​

Important

The TEI-XML file for the corpus MUST be named corpusHeader.xml.

The teiHeader contains one attribute and four main tags. Of the four main tags, all are required for LAUDATIO, but only one is required by the RNG schema:

  • type must be "CorpusHeader".
  • fileDesc (file description) provides description of a document (here: corpus), such as its title, editor(s), publisher, etc.
  • ‼️ profileDesc (text-profile description) Provides a detailed description of non-bibliographic aspects of a text, specifically the languages and sublanguages used, the situation in which it was produced, the participants and their setting.
  • ‼️ encodingDesc (encoding description) documents the relationship between an electronic text and the source or sources from which it was derived. For LAUDATIO: The element encodingDesc refers to the different formats the corpus is uploaded in. For each format, a single encodingDesc contains information about what kind of annotation (including keys and values) and what kind of formats were used.
  • ‼️ revisionDesc (revision description) summarizes the revision history for a file.

The basic structure of the corpus TEI header is thus as follows:

<?xml version='1.0' encoding='utf-8'?>
<?xml-model href="./corpus.rng" type="application/xml" schematypens="http://relaxng.org/ns/structure/1.0"?>
<TEI xmlns="http://www.tei-c.org/ns/1.0">
    <teiHeader type="CorpusHeader">
        <fileDesc>...</fileDesc>
        <profileDesc>...</profileDesc>
        <encodingDesc>...</encodingDesc>
        <revisionDesc>...</revisionDesc>
    </teiHeader>
    <text/>
</TEI>

The four main tags are described in detail below.



fileDesc​

The meaning of the tags under fileDesc is left untouched, but they simply reference the object Corpus. For example, the tag editor refers to the responsible compiler(s) of the corpus, author refers to e.g. the annotator(s) of the corpus, and title refers to the title of the entire corpus. Tags used within fileDesc are:

  • titleStmt (title statement) groups information about the title of a work and those responsible for its content.
  • ‼️ extent describes the approximate size of a text stored on some carrier medium or of some other object, digital or non-digital, specified in any convenient units.
  • publicationStmt (publication statement) groups information concerning the publication or distribution of an electronic or other text.
  • sourceDesc (source description) describes the source from which an electronic text was derived or generated, typically a bibliographic description in the case of a digitized text, or a phrase such as "born digital" for a text which has no previous existence.

The basic structure of fileDesc is as follows:

<fileDesc>
    <titleStmt>
        <title>...</title>
        <editor>...</editor>
        <author>...</author>
        <respStmt>...</respStmt>
    </titleStmt>
    <extent>...</extent>
    <publicationStmt>
        <authority>...</authority>
        <idno>...</idno>
        <availability>...</availability>
        <date>...</date>
    </publicationStmt>
    <sourceDesc>...</sourceDesc>
</fileDesc>

titleStmt​

The titleStmt contains the title, editors, annotators, and technical staff (infrastructure) of the corpus:

  • title contains the corpus title.
  • ‼️ editor denotes those responsible for corpus creation.
    • 0️⃣ n specifies the numerical order of people to be listed.
    • role specifies the role of this person. Only possible value is "CorpusEditor".
    • ‼️ persName (personal name) contains a proper noun or proper-noun phrase referring to a person, possibly including one or more of the person's forenames, surnames, honorifics, added names, etc.
      • 🔂 key is used to refer to an external authority file. Possible values are "orcid" and "gnd". If present, ref should be present as well. Note that this information is currently not ingested and shown in LAUDATIO.
      • 🔂 ref contains the URI to the external authority file entry. If present, key should be present as well. Note that this information is currently not ingested and shown in LAUDATIO.
      • 🔂 forename contains a forename, given or baptismal name. If present, surname should be present as well.
      • 🔂 surname contains a family (inherited) name, as opposed to a given, baptismal, or nick name. If present, forename should be present as well.
    • 🔂 affiliation contains an informal description of a person's present or past affiliation with some organization, for example an employer or sponsor. If at least one editor has the affiliation tag, all other editors should have it too. Plus, all affiliation tags must contain the same number of orgName tags.
      • 🔂 orgName (organization name) contains the name of the organization a person is affiliated with.
        • ‼️ type must be either "Department" or "Institution".
  • 0️⃣ author denotes the technical staff of the corpus (those responsible for annotation and infrastructure).
    • 0️⃣ n specifies the numerical order of people to be listed. These can be counted separately for each role below!
    • role specifies the role of this person. Possible values are "Annotator" and "Infrastructure".
    • 0️⃣ persName (personal name) contains a proper noun or proper-noun phrase referring to a person, possibly including one or more of the person's forenames, surnames, honorifics, added names, etc.
      • 🔂 key is used to refer to an external authority file. Possible values are "orcid" and "gnd". If present, ref should be present as well. Note that this information is currently not ingested and shown in LAUDATIO.
      • 🔂 ref contains the URI to the external authority file entry. If present, key should be present as well. Note that this information is currently not ingested and shown in LAUDATIO.
      • 🔂 forename contains a forename, given or baptismal name. If present, surname should be present as well.
      • 🔂 surname contains a family (inherited) name, as opposed to a given, baptismal, or nick name. If present, forename should be present as well.
    • 🔂 affiliation contains an informal description of a person's present or past affiliation with some organization, for example an employer or sponsor. If at least one author has the affiliation tag, all other authors should have it too. Plus, if present, all affiliation tags must contain the same number of orgName tags.
      • 🔂 orgName (organization name) contains the name of the organization a person is affiliated with.
        • ‼️ type must be either "Department" or "Institution".
  • 0️⃣ respStmt (responsibility statement) specifies other kinds of corpus responsibilities, e.g. those responsible for the metadata schema. Note that this information is currently not ingested and shown in LAUDATIO.
    • resp describes what the responsiblity entails (e.g. metadata).
    • 0️⃣ persName (personal name) contains a proper noun or proper-noun phrase referring to a person, possibly including one or more of the person's forenames, surnames, honorifics, added names, etc.
      • 🔂 key is used to refer to an external authority file. Possible values are "orcid" and "gnd". If present, ref should be present as well. Note that this information is currently not ingested and shown in LAUDATIO.
      • 🔂 ref contains the URI to the external authority file entry. If present, key should be present as well. Note that this information is currently not ingested and shown in LAUDATIO.
      • 0️⃣ forename contains a forename, given or baptismal name.
      • 0️⃣ surname contains a family (inherited) name, as opposed to a given, baptismal, or nick name.
      • 0️⃣ orgName (organization name) contains the name of the organization a person is affiliated with.
        • 0️⃣ type must be either "Department" or "Institution".

An example of the titleStmt looks like this:

<titleStmt>
    <title>Demo Corpus</title>
    <editor n="1" role="CorpusEditor"><!-- add more editors if necessary -->
        <persName><!-- optional, recommended add norm references such as OCRIDs as attributes, 
            add @key and @ref, e.g.: key="orcid" ref="https://orcid.org/1234-1234-1234-" -->
            <forename>Jane</forename>
            <surname>Doe</surname>
        </persName>
        <affiliation>
            <orgName type="Department">Department of Linguistics</orgName>
            <orgName type="Institution">University</orgName>
        </affiliation>
    </editor>
    <author n="1" role="Annotator"><!-- add more annotators if necessary, count in attribute @n -->
        <persName>
            <forename>John</forename>
            <surname>Doe</surname>
        </persName>
        <affiliation>
            <orgName type="Department">Department of History</orgName>
            <orgName type="Institution">University</orgName>
        </affiliation>
    </author>
    <author n="1" role="Infrastructure">
        <persName>
            <forename>Someone</forename>
            <surname>Else</surname>
        </persName>
        <affiliation>
            <orgName type="Department">Institut für deutsche Sprache und Linguistik</orgName>
            <orgName type="Institution">Humboldt-Universität zu Berlin</orgName>
        </affiliation>
    </author>
    <respStmt>
        <resp>Metadata</resp>
        <persName><!-- add more if necessary -->
            <forename>John</forename>
            <surname>Doe</surname>
        </persName>
        <orgName type="Department">Department of History</orgName>
        <orgName type="Institution">University</orgName>
    </respStmt>
</titleStmt>

extent​

The ‼️ extent provides information about the size of the corpus. The attribute type defines the counting unit, which can be "Tokens" or "Words".

  • ‼️ extent describes the approximate size of a text stored on some carrier medium or of some other object, digital or non-digital, specified in any convenient units.
    • type Characterizes the element in some sense, using any convenient classification scheme or typology. Possible values are "Tokens" or "Words".

An example for this tag is:

<extent type="Tokens">123456789</extent>

publicationStmt​

The publicationStmt covers the authority, availability, and publication dates.

  • authority (release authority) supplies the name of a person or other agency responsible for making a work available, other than a publisher or distributor.
  • 0️⃣ idno (identifier) supplies any form of identifier used to identify some object, such as a bibliographic item, a person, a title, an organization, etc. in a standardized way.
  • ‼️ availability supplies information about the availability of a text, for example any restrictions on its use or distribution, its copyright status, any licence applying to it, etc.
    • status supplies a code identifying the current availability of the text. Possible values are "free", "restricted", or "unknown", but should be "free" for LAUDATIO.
    • ‼️ licence information on the licence applying to the corpus.
      • target supplies a URI to the chosen licence.
    • ‼️ p (paragraph) shortly describes the licence.
  • ‼️ date contains a date in any format and refers to all corpus releases. This means that for each corpus release up until the current version, there must be a date tag.
    • when supplies the value of the date or time in a standard form, e.g. yyyy or yyyy-mm-dd.
    • 0️⃣ type specifies the type of release.

An example of publicationStmt is given here:

<publicationStmt>
    <authority>Hamburg University</authority>
    <!-- e.g. your university -->
    <idno>xyz</idno>
    <!-- add identificators if available -->
    <availability status="free">
        <licence target="http://creativecommons.org/licenses/by/4.0/" />
        <p>The corpus is published under a CC-BY 4.0 licence.</p>
        <!-- prose description of the licence -->
    </availability>
    <date type="CorpusRelease" when="2023-07-07">First complete corpus release.</date>
    <!-- short description of the release type -->
</publicationStmt>

sourceDesc​

sourceDesc contains a list of documents of the corpus, which are given by their IDs (see notes on xml:id of the document metadata).

  • list contains any sequence of items organized as a list.
    • type Type of list for the corpus. Only accepted value is "CorpusDocument".
    • item contains one component of a list (i.e. corpus document).
      • corresp The identifier for a document based on the ID given in the document TEI header.
      • n (number) gives a number (or other label) for an element, which is not necessarily unique within the document. This is simply an incrementing document counter.

An example is given here:

<sourceDesc>
    <list type="CorpusDocument">
        <!-- each document header contains an ID in <fileDesc xml:id="document1">, list the references here -->
        <item corresp="document1" n="1" />
        <item corresp="document2" n="2" />
    </list>
</sourceDesc>


profileDesc​

Important

Include at most three language tags and make sure to use each style only once!

The ‼️ profileDesc only contains metadata concerning the language of the corpus data including remarks on language type and lanuage area.

  • langUsage (language usage) describes the languages, sub-languages, registers, dialects, etc. represented within a text.
    • language characterizes a single language or sublanguage used within a text.
      • style contains an expression in some formal style definition language which defines the rendering or presentation used for this element in the source text; possible values: "Language", "LanguageType", "LanguageArea".
      • ident (identifier) supplies a language code constructed as defined in BCP 47 which is used to identify the language documented by this element, and which is referenced by the global xml:lang attribute.1

A concrete example of profileDesc would be:

<profileDesc>
    <langUsage>
        <language ident="de" style="Language">Early New High German</language>
        <language ident="de" style="LanguageArea">Southern dialects</language>
        <language ident="de" style="LanguageType">Bavarian</language>
    </langUsage>
</profileDesc>


encodingDesc​

The ‼️ encodingDesc contains descriptions of the corpus annotation guidelines for each format the corpus is provided in. A corpus header can therefore have more than one encodingDesc. It has the following attributes and tags:

  • 0️⃣ n (number) gives a number (or other label) for an element, which is not necessarily unique within the document. This is simply an incrementing counter.
  • 0️⃣ style describes the kind of format. It is not ingested or shown in LAUDATIO.
  • ‼️ appInfo specifies the corpus format.
  • ‼️ projectDesc describes the research project and its references.
  • ‼️ editorialDecl is used to describe how the corpus was segmented and normalized.
  • ‼️ tagsDecl gives the actual annotation layers that this corpus format includes and which are roughly divided according to their functions.

The basic structure is thus as follows:

<encodingDesc n="1">
    <!-- each <encodingDesc> describes the annotation layers that are released in a format of the corpus;
        add more <encodingsDesc> if the corpus has more than one format -->
    <appInfo>...</appInfo>
    <projectDesc>...</projectDesc>
    <editorialDecl>...</editorialDecl>
    <tagsDecl>...</tagsDecl>
</encodingDesc>

appInfo​

The ‼️ appInfo (application information) records information about the corpus format. While it is optional according to the schema, it must be given in order to correctly assign uploaded corpus data in LAUDATIO.

The elements within ‼️ appInfo are:

  • application provides information about an used format.
    • 0️⃣ n (number) gives a number (or other label) for an element, which is not necessarily unique within the document. This is simply an incrementing counter.
    • 0️⃣ style is a free text field. This is not ingested/shown in LAUDATIO.
    • 0️⃣ subtype is a free text field. This is not ingested/shown in LAUDATIO.
    • ident supplies an identifier for the format (independent of its version number or display name), e.g. "ANNIS", "TXT", or "EXMARaLDA".
    • version supplies a version number for the format (independent of its identifier or display name), e.g. "3.0".
    • label contains a free-form field to name/describe the format.
<appInfo>
    <application ident="EXMARaLDA" version="3.0">
        <label>EXMARaLDA XML for Partitur editor.</label>
    </application>
</appInfo>

projectDesc​

‼️ projectDesc (project description) describes in detail the aim or purpose for which an electronic file was encoded, together with any other relevant information concerning the process by which it was assembled or collected.

  • p (paragraph) contains the description of the corpus.
    • 0️⃣ ref (reference) defines a reference to another location (e.g. the project homepage), possibly modified by additional text or comment.
      • 0️⃣ target specifies the destination of the reference by supplying one or more URI References
<projectDesc>
    <p>
        <ref target="www.project-page.com"/>Data annotation was carried out in our project: project description.
    </p>
</projectDesc>

editorialDecl​

‼️ editorialDecl (editorial practice declaration) provides details of editorial principles and practices applied during the encoding of a text.

  • ‼️ segmentation describes the principles according to which the text has been segmented, for example into sentences, tone-units, graphemic strata, etc.
    • p (paragraph) contains the description of the segmentation.
  • 0️⃣ normalization indicates the extent of normalization or regularization of the original source carried out in converting it to electronic form. This is not ingested/shown in LAUDATIO.2
    • p (paragraph) contains the description of the normalization.
<editorialDecl>
    <segmentation>
        <p>Annotation 'dipl' has an independent segmentation. Every other annotation is
            based on the segmentation of 'dipl.</p>
    </segmentation>
    <normalization>
        <p>No normalization is applied.</p>
    </normalization>
</editorialDecl>

tagsDecl​

‼️ tagsDecl (tagging declaration) provides detailed information about the tagging applied to a document. These are the annotation layers.

  • ‼️ namespace supplies the formal name of the namespace to which the elements documented by its children belong.
    • ‼️ xml:id is the identifier for the annotation layer,. It should be the same as the annotation header file name and the attribute corresp of title of the annotation metadata.
    • name specifies the full formal name of the namespace concerned.
    • rend (rendition) indicates how the element in question was rendered or presented in the source text. The annotation layers are classified with a closed list of categories:
      • "Transcription"
      • "Lexical"
      • "Morphological"
      • "Syntactic"
      • "Graphical"
      • "MarkUp"
      • "Meta"
      • "Other"
    • tagUsage supplies information about the usage of a specific element within a text. These are the possible annotation values for a given layer.
      • gi (element name) specifies the name (generic identifier) of the element indicated by the tag.
<tagsDecl>
    <namespace xml:id="dipl" name="dipl" rend="Transcription">
        <tagUsage gi="String">Diplomatic,character based transcription.</tagUsage>
    </namespace>
    <namespace xml:id="pos" name="POS" rend="Lexical">
        <tagUsage gi="DET">Determiner.</tagUsage>
        <tagUsage gi="N">Noun.</tagUsage>
        <tagUsage gi="V">Verb.</tagUsage>
        <tagUsage gi="P">Punctuation.</tagUsage>
        <tagUsage gi="PRON">Pronoun.</tagUsage>
    </namespace>
</tagsDecl>


revisionDesc​

info

The last change element determines the current corpus version! In the example below, this corpus would be uploaded in version "3.0".

‼️ revisionDesc refers to the version history of the whole corpus. Corpora in the LAUDATIO repository may have several versions. The changes between these versions will be provided here.

  • ‼️ revisionDesc (revision description) summarizes the revision history for a file.
    • change documents a change or set of changes made during the production of a source document, or during the revision of an electronic file. A text within the tag should also be provided!
      • ‼️ n (number) gives a number (or other label) for an element, which is not necessarily unique within the document. Note that this is not a counter, but means the corpus version numbers, e.g. "1.0".
      • ‼️ when supplies the value of the date or time in a standard form, e.g. yyyy or yyyy-mm-dd.
      • 0️⃣ type specifies the kind of corpus revision undertaken. This is not ingested/shown in LAUDATIO.
      • 0️⃣ who indicates the person, or group of people, to whom the element content is ascribed. This is not ingested/shown in LAUDATIO.
<revisionDesc>
    <change n="1.0" type="CorpusRelease" when="2011" who="NA">First release of the corpus.</change>
    <change n="2.0" type="QualityCheck" when="2012-05-31" who="LAUDATIO">Consistency check of
        annotation keys. Change of corpus architecture, multiple
        tokenizations/segmentations.
    </change>
    <change n="3.0" type="CorpusExtension" when="2013-05-15" who="LAUDATIO">Consistency check of
        annotation keys. Change of corpus architecture, multiple tokenizations/segmentations.
        Additional texts were added to the corpus. Several annotation keys were deleted, replaced
        and checked.
    </change>
</revisionDesc>

Footnotes​

  1. BCP 47 seems to be interoperable with ISO 639 language tags, so this means we could work with very granular language tags. ↩

  2. However, there is a table column for it in LAUDATIO... ↩