Metadata

The simple example included a way to index embedded metadata. Let's say this is our input file:

<?xml version="1.0" ?>
<root>
    <document>
        <text>
            <!-- ... document contents... -->
        </text>
        <metadata id='1234'>
            <meta name='title'>How to configure indexing</meta>
            <meta name='author'>Jan Niestadt</meta>
            <meta name='description'>Shedding some light on this indexing business!</meta>
        </metadata>
    </document>
</root>

To configure how metadata should be indexed, you can either name each metadata field you want to index separately, or you can use forEachPath to index a number of similar elements as metadata:

## Embedded metadata in document
metadata:

  # What element contains the metadata (relative to documentPath)
- containerPath: metadata

  # What metadata fields do we have?
  fields:

    # <metadata/> tag has an id attribute we want to index as docId
  - name: docId
    valuePath: "@id"

    # Each <meta/> child element of <metadata/> corresponds with a metadata field
  - forEachPath: meta
    namePath: "@name"   # name attribute contains field name
    valuePath: .        # element text is the field value

It's also possible to process metadata values before they are indexed (see Processing values), although it's often preferable to do as much processing as possible in XPath.

As you can see, metadata is a list, so you can define several metadata blocks, each with their own containerPath.

Nested metadata blocks

In-development feature

Try this feature on dev (Docker image or branch). It will be finalized for version 5.0.

You can even nest metadata blocks, so you can use multiple levels of containerPaths:

metadata:
  - containerPath: //metadata
    blocks:
      - containerPath: author        # relative to //metadata
        fields:
          - name: authorName
            valuePath: name          # relative to //metadata/author
          - name: authorYearOfBirth
            valuePath: yearOfBirth   # relative to //metadata/author
      - containerPath: title         # relative to //metadata
        fields:
          - name: titleLevel1
            valuePath: main          # relative to //metadata/title
          - name: titleLevel2
            valuePath: sub           # relative to //metadata/title

As you can see, this can help reduce duplication, keeping your XPath expressions short and readable.

Tokenize or not?

By default, metadata fields are tokenized, but it can sometimes be useful to index a metadata field without tokenizing it. One example of this is a field containing a code such as ABC.123: if these contain characters that normally would indicate a token boundary, like a period (.) , the code would be split into several tokens, which is usually not what you want.

To prevent a metadata field from being tokenized:

metadata:

- containerPath: metadata

  fields:

    # This field should not be split into words
  - name: code
    valuePath: @code
    type: untokenized

The persistent identifier field (marked with role: pid) will automatically be set to untokenized.

Numeric fields

To index a numeric field (currently supports only integer values):

metadata:
- containerPath: metadata
  fields:
  - name: year
    valuePath: publication/year
    type: numeric

We may consider adding other specific field types (floating point, date, vector) in the future.

Linking to external document metadata

Old linkedDocuments feature

Previously, external metadata could be indexed using the complex linkedDocuments feature. This was removed after 4.x.

The doc() approach here is a standard XPath technique and is significantly easier to use. It is available from dev/5.x but may not work properly in older versions.

If your metadata is stored in separate files, you can use the XPath doc() function to load the metadata file and extract the relevant information.

For example, if your document looks like this:

<?xml version="1.0" ?>
<document id="12345">
    <text>
        <!-- ... document contents... -->
    </text>
</document>

And the metadata file metadata/12345.xml looks like this:

<?xml version="1.0" ?>
<metadata>
    <title>How to configure indexing</title>
    <author>Jan Niestadt</author>
</metadata>

You can configure the metadata indexing like this:

## Embedded metadata in document
metadata:
    # What element contains the metadata (relative to documentPath)
    # (but here we actually point to an external file using doc())
  - containerPath: doc(concat('metadata/', ./@id, '.xml'))
    
    # What metadata fields do we have?
    fields:
    
        # Load metadata from external file using doc()
    - name: title
      valuePath: ./metadata/title
    
    - name: author
      valuePath: ./metadata/author

Schemes

By default, doc() looks for files on the local filesystem.

However, you can prefix the path with a scheme to load files from other sources. For example, use https://example.com/some/path.xml to load from a web server.

archive scheme

In-development feature

Try this feature on dev (Docker image or branch). It will be finalized for version 5.0.

You can also use archive: to load a file from an archive, for example:

containerPath: doc(concat('archive:metadata.zip/', ./@id, '.xml'))

Custom schemes

In-development feature

Try this feature on dev (Docker image or branch). It will be finalized for version 5.0.

You can even add your own schemes. For example, my-db:12345 might load a document from a database. Each scheme such as archive refers to a IndexSourceType plugin, and adding one isn't difficult.

Using the original input file path

In-development feature

Try this feature on dev (Docker image or branch). It will be finalized for version 5.0.

You can use $inputFilePath from XPath if you need. For example, if your input file is content/doc0123.xml and you want to link to metadata/meta0123.xml, you could use this XPath:

containerPath: doc(concat('metadata/meta', replace($inputFilePath, '^.*doc(\d+)\.xml$', '$1'), '.xml'))

Store (part of) a linked document

If you need to store the entire metadata XML content, this should work:

metadata:
- containerPath: doc(concat('metadata/', ./@id, '.xml'))
  fields:
    - name: metadata-xml
      valuePath: serialize(.)

Custom properties

Note that custom properties may be removed in a future version.

Just like with annotations, you can specify a displayName, description and uiType for a metadata field. This information is not used by BlackLab, but can be used by BlackLab Frontend or another application. For example, see Metadata (Filters)

In the fields section, you can specify uiType for each field to override the default GUI widget to use for the field. By default, fields that have only a few values will use select, while others will use text. There's also a range type for a range of numbers.

Example:

metadata:
- fields:
    - name: author
      uiType: select
      
    - name: year
      uiType: range
      
    - name: genre
      uiType: text

Again, note that these properties may be removed from the .blf.yaml file specification in the future. It makes more sense to configure the frontend directly, for example using a custom script. See Customizing the interface.

Fixed metadata value

You can add a field with a fixed value to every document indexed. This could be useful if you plan to add several data sets to one index and want to make sure each document is tagged with the data set name. To do this, simply specify value instead of valuePath.

metadata:

- containerPath: metadata

  fields:

    # Regular metadata field    
  - name: author
    valuePath: author

    # Metadata field with fixed value
  - name: collection
    value: blacklab-docs

Persistent document identifiers (PIDs)

To ensure document ids in URLs and responses don't change when you re-index, you should designate one of your metadata fields as the persistent identifier (PID). This should be unique among documents. E.g. it might be a unique number or code that can be used to refer to the document.

Tag this metadata field with role: pid so BlackLab knows to use it as the persistent identifier.

metadata:

- containerPath: metadata

  fields:   
  - name: id
    valuePath: @id
    role: pid

Corpus metadata

Each BlackLab corpus has its own metadata, recording information such as the time the index was generated and the BlackLab version used, plus information about annotations and metadata fields.

Some of this information is generated as part of the indexing process, and some of the information is copied directly from the input format configuration file if specified. This information is mostly used by applications to learn about the structure of the corpus, get human-friendly names for the various parts, and decide what UI widget to show for a metadata field.

The best way to influence the corpus metadata is by including a special section corpusConfig in your format configuration file. This section may contains certain settings to be copied directly into the index file when it is created:

    # The settings in this block will be copied into indexmetadata.yaml
    corpusConfig:
  
      # Some basic information about the corpus that may be used by a user interface.
      displayName: OpenSonar              # Corpus name to display in user interface
      description: The OpenSonar corpus.  # Corpus description to display in user interface
      contentViewable: false              # Is the user allowed to view whole documents? [false]
      textDirection: LTR                  # What's the text direction of this corpus? [LTR]

      # Metadata fields with a special meaning
      specialFields:
        titleField: title      # used to display document title in interface
        authorField: author    # used to display author in interface
        dateField: date        # used to display document date in interface
      
      # How to group metadata fields in user interface
      metadataFieldGroups:
      - name: First group      # Text on tab, if there's more than one group
        fields:                # Metadata fields to display on this tab
        - author
        - title
      - name: Second group
        fields:
        - date
        - keywords

If you add addRemainingFields: true to one of the groups, any field that wasn't explicitly listed will be added to that group.

There's also a complete annotated index metadata file if you want to know more details about that.

There are also (hacky) ways to make changes to the corpus metadata after it was indexed: you can export the metadata to a file and re-import it later (older corpora had an external indexmetadata.yaml file that could be edited directly). Start the IndexTool with --help to learn more, but be careful, as it is easy to make the index unusable this way.

Metadata on fragments

In-development feature

Try this feature on dev (Docker image or branch). It will be finalized for version 5.0.

Advanced feature

This is an advanced feature that is probably not needed for most corpora.

It is possible to annotate and index parts of your documents with metadata. In BlackLab, we call these parts "(document) fragments". A fragment inherits the document metadata but can also add its own or override values.

This can be useful if, for example, your historical corpus contains documents where different parts of it were written by different authors or in different years. In this case, you want to be able to search for text written by a specific author or in a specific year, but you don't want to split the document into multiple documents, because that would lose the context of the text. Fragments make this possible.

Differences from spans

Fragments differ from spans (inline tags such as <s/> or <p/> in a few ways:

• spans have a type (e.g. s or p), fragments do not.
• spans can have textual attributes that you can search on using regular expressions, but these cannot be tokenized. fragment metadata works the same as document metadata, so it can be tokenized and searched using the same syntax (Lucene query language). It also supports numeric fields.
• when indexing fragments normally inherit document metadata, and nested fragments inherit metadata from their parent fragment. Spans do not inherit anything.

Fragments and search

If you search for a word in text written by a specific author, that word will only be found in matching fragments. The match will not indicate the fragment in any way though, only the document it was found in. If you want to know which fragment it was found in, you should index fragments as both fragment and span (configure both separately), so you can use within <frag /> to find the fragment that contains the match.

When searching for documents (e.g. /docs operation in BlackLab Server), only full documents will be returned, even if you filtered by fragment metadata.

Note that if your metadata filter matches two adjacent fragments, you will also find matches that cross the fragment boundary. This is unlikely to be a major issue, but if you want to make sure this cannot happen, you can again index the fragment as a span as well and use the within operator to restrict your search to a single fragment.

Example

Let's look at how to configure indexing fragments.

Below are two examples of a different style of defining fragments: using inline tags, and using standoff annotations.

An example of using inline tags to define fragments. The <metadata/> elements are used to mark fragments of the document. The id, year, and author attributes are metadata for that fragment.

The input document:

<?xml version="1.0" ?>
<doc>
    <metadata title="Test title" id="doc-01" author="Katrien" />
    <text>
        <metadata id="doc-01-frag-01" year="2025">
            <s>
                <w>This</w>
                <w>is</w>
                <w>a</w>
                <w>fragment</w>.
            </s>
        </metadata>
        <metadata id="doc-01-frag-02" year="2026" author="Jesse">
            <s>
                <w>Here's</w>
                <w>another</w>
                <w>one</w>.
            </s>
        </metadata>
        <s>
            <w>One</w>
            <w>more</w>!
        </s>
    </text>
</doc>

To index this input document, use type: fragment for the inlineTags entry:

# What element starts a new document?
# (the only absolute XPath; the rest is relative)
documentPath: /doc

# Annotated, CQL-searchable fields.
# We usually have just one, named "contents".
annotatedFields:
    contents:
        containerPath: text # containerPath for the contents field (relative to documentPath)
        wordPath: .//w
        annotations:
            - name: word
              valuePath: .

        inlineTags:
            - path: .//s
            - path: .//metadata
              type: fragment
              # (optional) path to the actual metadata container (relative to the element matched by 'path')
              #metadataContainerPath: .

# How to index metadata for documents and fragments
metadata:

    # (this is the document-level metadata container, relative to documentPath.
    #  this is ignored for fragments: the fragment is its own metadata container)
    containerPath: metadata

    fields:
        - name: id
          valuePath: "@id"
          role: pid
          # "fragments: separate" indicates that id should NOT inherit to fragment;
          # (this is already implied by "role: pid", so specifying it here is superfluous)
          fragments: separate
        - name: title
          valuePath: "@title"
        - name: author
          valuePath: "@author"
        - name: year
          valuePath: "@year"

Missing fragment?

Notice that in the examples above, two fragments are defined, from A to B and from B to C. But from milestone C to D there is no fragment defined. In this case, BlackLab will automatically create a fragment for that part of the text, and it will inherit the document metadata.

Metadata field behavior

A metadata field is normally inherited from the document level in each fragment, unless the fragment overrides its value, of course. However, you can change this for a metadata field using the fragments setting. The possible values are:

• default (or omit the setting): fields that occur at the fragment level will ONLY be indexed in fragments. If the field also occurs at the document level in the input file (such as author in the example above), it will NOT be indexed at the document level, but instead the value specified there will inherit to any fragment that doesn't define its own author. In other words: in the example doc, each fragment without an author will index Katrien as its author. Searching for an author will only find fragment(s), never the full document.
• separate: with this behavior, document level and fragment level don't affect each other at all. The field will simply be indexed where it occurs in the input file. The field tagged with role: pid (id in the example) will automatically get this fragments setting. In the example, values for id are indexed at both the document level and at the fragment level, with each fragment having a unique id. The (automatically created) fragment from milestone C to D will not get the field id at all.
• docvalue: fields with this behavior will index at the document level and index the exact same value at the fragment level, ignoring the possibility of overriding it with a different value. Slightly speeds up indexing because we have fewer field rules to apply for each fragment.

Mostly, you should not need the fragments setting and leave it at the default behaviour.

Fragment metadata rules

Generally, it's best if documents and fragments have the same metadata structure in your input files, but that may not always be possible. You can deal with this by defining additional metadata rules for fragments.

For example, if the author field is stored in a different attribute at the fragment level (e.g. not author but frag-author), you could specify that as shown below.

    # Define the fragments and their metadata
    inlineTags:
      - path: .//metadata
        type: fragment
        # (optional) path to the actual metadata container (relative to the element matched by 'path')
        #metadataContainerPath: .

        # This section is entirely optional.
        # You may define additional metadata rules for fragments here if needed.
        # The document level rules are always applied, unless you set applyDocRules: false.
        metadata:
          - applyDocRules: true # (default value, can be omitted)
            containerPath: .    # (default value, can be omitted)
            fields:
              - name: author
                valuePath: "@frag-author"