TEI-LAUDATIO Transformation

We use the following XPath mapping file to extract the given metadata into a format compatible with Elasticseach:

{
  "//def:TEI/def:teiHeader/@style": "document_genre",
  "//def:TEI/def:teiHeader/def:fileDesc/@xml:id": "document_id",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:title/text()": "document_title",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author/def:persName/def:forename/text()": "document_author_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:author/def:persName/def:surname/text()": "document_author_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:persName/def:forename/text()": "document_editor_forename",
  "//def:TEI/def:teiHeader/def:fileDesc/def:titleStmt/def:editor/def:persName/def:surname/text()": "document_editor_surname",
  "//def:TEI/def:teiHeader/def:fileDesc/def:extent/@type": "document_size_type",
  "//def:TEI/def:teiHeader/def:fileDesc/def:extent/text()": "document_size_extent",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:schemaSpec/def:elementSpec/def:valList/def:valItem/@corresp": "document_list_of_annotations_id",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:schemaSpec/def:elementSpec/@ident": "document_list_of_annotations_type",
  "//def:TEI/def:teiHeader/def:encodingDesc/def:schemaSpec/def:elementSpec/def:valList/def:valItem/@ident": "document_list_of_annotations_name",
  "//def:TEI/def:teiHeader/def:profileDesc/def:langUsage/def:language/@style": "document_languages_style",
  "//def:TEI/def:teiHeader/def:profileDesc/def:langUsage/def:language/text()": "document_languages_language",
  "//def:TEI/def:teiHeader/def:profileDesc/def:langUsage/def:language/@ident": "document_languages_iso_code",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:pubPlace/text()": "document_publication_place",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:publisher/text()": "document_publication_publisher",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/@when": "document_publication_publishing_date",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:date/text()": "document_publication_date_information",
  "//def:TEI/def:teiHeader/def:fileDesc/def:publicationStmt/def:biblScope/text()": "document_publication_pages",
  "//def:TEI/def:teiHeader/def:fileDesc/def:seriesStmt/def:title/text()": "document_publication_series",
  "//def:TEI/def:teiHeader/def:fileDesc/def:seriesStmt/def:biblScope/@volume": "document_publication_volume",
  "//def:TEI/def:teiHeader/def:fileDesc/def:seriesStmt/def:editor/text()": "document_publication_editor",
  "//def:TEI/def:teiHeader/def:fileDesc/def:seriesStmt/def:biblScope/text()": "document_publication_volume_pages",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:msIdentifier/def:msName/text()": "document_history_title",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:history/def:origin/def:origDate/@notAfter-custom": "document_history_not_after",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:history/def:origin/def:origDate/@notBefore-custom": "document_history_not_before",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:history/def:origin/def:origDate/text()": "document_history_original_date",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:history/def:origin/def:origPlace/text()": "document_history_original_place",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:history/def:origin/def:title/text()": "document_history_original_title",
  "//def:TEI/def:teiHeader/def:fileDesc/def:sourceDesc/def:msDesc/def:history/def:origin/def:locus/text()": "document_history_location_in_manuscript",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/@n": "revision_document_version",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/@when": "revision_publishing_date",
  "//def:TEI/def:teiHeader/def:revisionDesc/def:change/text()": "revision_description"
}

After uploading a metadata header XML file, our backend first checks if the file is well-formed and valid. The transformation then happens by parsing both the XPath mapping file and the XML file to ingest the values like this:

from lxml import etree
import json

ingestiondict = {}

# Load xpath schema
with open(xpath) as f:
    xpathdict = json.load(f)

# Load XML document
with open(xml_file, "rb") as f:
    header = etree.parse(f)

for xpath_string in xpathdict:
    value = header.xpath(xpath_string, namespaces={"def": "http://www.tei-c.org/ns/1.0"})
    ingestiondict[xpathdict[xpath_string]] = value # we slightly process the value before ingestion, but leave this out here for simplicity 

The resulting JSON dictionary looks like this:

{
    'document_genre': ['City Chronicles'],
    'document_id': ['Mastro1_Chronik_aus_Kaiser_Sigmunds_Zeit_Nuernberg'],
    'document_title': ["Edition von: Chronik aus Kaiser Sigmund's Zeit bis 1434, mit Fortsetzung bis 1441"],
    'document_author_forename': ['NN'],
    'document_author_surname': ['NN'],
    ...
}

After some additional enrichment of the JSON dictionary (e.g. adding merged fields, relations (e.g. "has_annotations" for documents), adding the corpus Postgres database ID, or adding publication status), the dictionary is simply used as a document to index in Elasticsearch:

from elasticsearch import Elasticsearch

index = "documents"

es = Elasticsearch(...)
es.index(index=index, document=ingestiondict)