TEI-LAUDATIO Transformation

We use the following XPath mapping file to extract the given metadata into a format compatible with Elasticseach:

{
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:title/text()": "preparation_title",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:title/@corresp": "preparation_annotation_id",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:persName/def:forename/text()": "corpus_editor_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:persName/def:surname/text()": "corpus_editor_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:affiliation/def:orgName[@type='Department']/text()": "corpus_editor_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:affiliation/def:orgName[@type='Institution']/text()": "corpus_editor_affiliation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:persName/def:forename/text()": "preparation_author_annotator_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:persName/def:surname/text()": "preparation_author_annotator_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:affiliation/def:orgName[@type='Department']/text()": "preparation_author_annotator_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Annotator']/def:affiliation/def:orgName[@type='Institution']/text()": "preparation_author_annotator_affiliation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:persName/def:forename/text()": "preparation_author_infrastructure_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:persName/def:surname/text()": "preparation_author_infrastructure_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:affiliation/def:orgName[@type='Department']/text()": "preparation_author_infrastructure_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Infrastructure']/def:affiliation/def:orgName[@type='Institution']/text()": "preparation_author_infrastructure_affiliation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Transcription']/def:persName/def:forename/text()": "preparation_author_transcription_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Transcription']/def:persName/def:surname/text()": "preparation_author_transcription_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Transcription']/def:affiliation/def:orgName[@type='Department']/text()": "preparation_author_transcription_affiliation_department",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author[@role = 'Transcription']/def:affiliation/def:orgName[@type='Institution']/text()": "preparation_author_transcription_affilitation_institution",
  "//def:TEI/def:teiHeader/def:fileDesc/def:extent/@type": "preparation_size_type",
  "//def:TEI/def:teiHeader/def:fileDesc/def:extent/text()": "preparation_size_value",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/@type": "preparation_publication_release_type",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/@when": "preparation_publication_release_date",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/text()": "preparation_publication_license_description",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:p/text()": "preparation_publication_tag_description",
  "//def:TEI/def:teiHeader/def:encodingDesc/@style": "preparation_encoding_step",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/@style": "preparation_encoding_models",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@ident": "preparation_encoding_tool",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/def:label/text()": "preparation_encoding_full_name",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/def:p/text()": "preparation_encoding_description",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@style": "preparation_encoding_file_extension",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@version": "preparation_encoding_tool_version",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@type": "preparation_encoding_annotation_group",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:appInfo/def:application/@subtype": "preparation_encoding_annotation_sub_group",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:editorialDecl/def:segmentation/@style": "preparation_encoding_segmentation_style",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:editorialDecl/def:segmentation/@corresp": "preparation_encoding_segmentation_type",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:editorialDecl/def:segmentation/def:p/text()": "preparation_encoding_segmentation_description",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:projectDesc/def:p/text()": "preparation_encoding_project_url",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/@n": "preparation_revision_corpus_version",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/@when": "preparation_revision_publishing_date",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/text()": "preparation_revision_description"
}

After uploading a metadata header XML file, our backend first checks if the file is well-formed and valid. The transformation then happens by parsing both the XPath mapping file and the XML file to ingest the values like this:

from lxml import etree
import json

ingestiondict = {}

# Load xpath schema
with open(xpath) as f:
    xpathdict = json.load(f)

# Load XML document
with open(xml_file, "rb") as f:
    header = etree.parse(f)

for xpath_string in xpathdict:
    value = header.xpath(xpath_string, namespaces={"def": "http://www.tei-c.org/ns/1.0"})
    ingestiondict[xpathdict[xpath_string]] = value # we slightly process the value before ingestion, but leave this out here for simplicity 

The resulting JSON dictionary looks like this:

{
    'preparation_title': ['text'],
    'preparation_annotation_id': ['text'],
    'corpus_editor_forename': ['Torsten', 'Carolin'],
    'corpus_editor_surname': ['Hiltmann', 'Odebrecht'],
    ...
}

After some additional enrichment of the JSON dictionary (e.g. adding merged fields, relations (e.g. "in_documents" for annotations), adding the corpus Postgres database ID, or adding publication status), the dictionary is simply used as a document to index in Elasticsearch:

from elasticsearch import Elasticsearch

index = "annotations"

es = Elasticsearch(...)
es.index(index=index, document=ingestiondict)