Skip to main content

TEI-LAUDATIO Transformation

We use the following XPath mapping file to extract the given metadata into a format compatible with Elasticseach:

{
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:title/text()": "corpus_title",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:persName/def:forename/text()": "corpus_editor_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:persName/def:surname/text()": "corpus_editor_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:affiliation/def:orgName[@type='Department']/text()": "corpus_editor_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:affiliation/def:orgName[@type='Institution']/text()": "corpus_editor_affiliation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:persName/def:forename/text()": "corpus_annotator_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:persName/def:surname/text()": "corpus_annotator_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:affiliation/def:orgName[@type='Department']/text()": "corpus_annotator_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:affiliation/def:orgName[@type='Institution']/text()": "corpus_annotator_affiliation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:persName/def:forename/text()": "corpus_author_infrastructure_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:persName/def:surname/text()": "corpus_author_infrastructure_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:affiliation/def:orgName[@type='Department']/text()": "corpus_infrastructure_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:affiliation/def:orgName[@type='Institution']/text()": "corpus_infrastructure_affiliation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:extent/@type": "corpus_size_type",
  "//def:TEI/def:teiHeader/def:fileDesc/def:extent/text()": "corpus_size_value",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:authority/text()": "corpus_publication_publisher",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:availability/def:licence/@target": "corpus_publication_license",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:availability/def:p/text()": "corpus_publication_license_description",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/@when": "corpus_publication_publication_date",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/text()": "corpus_publication_publication_description",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:list/def:item/@corresp": "corpus_documents",
  "//def:TEI/def:teiHeader/def:profileDesc/def:langUsage/def:language/@style": "corpus_languages_style",
  "//def:TEI/def:teiHeader/def:profileDesc/def:langUsage/def:language/text()": "corpus_languages_language",
  "//def:TEI/def:teiHeader/def:profileDesc/def:langUsage/def:language/@ident": "corpus_languages_iso_identifier",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@ident": "annotation_encoding_format",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@version": "annotation_encoding_version",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/def:label/text()": "annotation_encoding_tool",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:projectDesc/def:p/def:ref/@target": "corpus_encoding_project_homepage",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:projectDesc/def:p/text()": "corpus_encoding_project_description",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:editorialDecl/def:segmentation/def:p/text()": "annotation_encoding_segmentation",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/@n": "corpus_version",
  "//def:change[last()]/@n": "publication_version",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/@when": "corpus_version_publishing_date",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/text()": "corpus_version_description",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:tagsDecl/def:namespace/@id": "annotation_id"
}
  • Note that the last transformation, the mapping to "annotation_id", is not used in LAUDATIO.

After uploading a metadata header XML file, our backend first checks if the file is well-formed and valid. The transformation then happens by parsing both the XPath mapping file and the XML file to ingest the values like this:

from lxml import etree
import json

ingestiondict = {}

# Load xpath schema
with open(xpath) as f:
    xpathdict = json.load(f)

# Load XML document
with open(xml_file, "rb") as f:
    header = etree.parse(f)

for xpath_string in xpathdict:
    value = header.xpath(xpath_string, namespaces={"def": "http://www.tei-c.org/ns/1.0"})
    ingestiondict[xpathdict[xpath_string]] = value # we slightly process the value before ingestion, but leave this out here for simplicity

The resulting JSON dictionary looks like this:

{
    'corpus_title': ['RIDGES Herbology'],
    'corpus_editor_forename': ['Anke', 'Carolin', 'Thomas', 'Gohar', 'Catharina'],
    'corpus_editor_surname': ['Lüdeling', 'Odebrecht', 'Krause', 'Schnelle', 'Fischer'],
    ...
}

After some additional enrichment of the JSON dictionary (e.g. adding merged fields, relations (e.g. "has_documents" for corpora), adding the corpus Postgres database ID, or adding publication status), the dictionary is simply used as a document to index in Elasticsearch:

from elasticsearch import Elasticsearch

index = "corpora"

es = Elasticsearch(...)
es.index(index=index, document=ingestiondict)