Frequently Asked Questions

Here are some questions that may or may not have been asked frequently. As a bonus, we've even thrown in some answers.

If your question is not here, or the answer didn't satisfy, please contact us. We're always happy to hear from you.

Troubleshooting

Corpus compatibility?

For information about indexed corpus compatibility ("does an older corpus work with a newer BlackLab, or do I have to reindex?", see the downloads page.

No words in my files?

BlackLab needs tokenized input files. This means the word boundaries have already been determined so BlackLab can just index words as it parses the file.

If your input files are not tokenized, BlackLab may complain that it didn't find any words. This can also happen if your input format configuration doesn't match your input data. See how to configure indexing or contact us.

Why is BlackLab slow?

This can have many causes:

• Your memory configuration may not be optimal.

• Your machine specifications may be insufficient for what you're trying to do. As a rough indication: we run a 4.5 billion token corpus on a (virtual) server with 50 GB of memory and 16 cores, with few simultaneous users that generally perform simple queries, with the occasional heavier query. This machine could probably handle heavier use than this.

• Certain types of BCQL queries can also be quite taxing on the machine. If you encounter an unexpectedly slow query, let us know and we'll see if we can help.

View whole documents?

You may be getting "permission denied" or such when trying to view a whole document in BlackLab Frontend or via BlackLab Server. This is due to the corpusConfig.contentViewable setting in the indexing configuration file.

See the note about contentViewable on the BlackLab Server overview page.

Special chars in URLs?

If you're not using Docker, you have to configure Tomcat to use Unicode URLs.

To ensure the correct handling of accented characters in (search) URLs, you should configure Tomcat to interpret URLs as UTF-8 (by default, it does ISO-8859-1) by adding an attribute URIEncoding="UTF-8" to the <Connector/> element with the attribute port="8080" in Tomcat's server.xml file.

Of course, make sure that URLs you send to BlackLab are URL-encoded using UTF-8 (so e.g. searching for "señor" corresponds to a request like http://myserver/blacklab-server/mycorpus/hits?patt=%22se%C3%B1or%22). BlackLab Frontend does this by default.

About the project

Who is BlackLab for?

BlackLab is a good choice if you want to search a large body of text annotated with extra information per word (e.g. lemma, part of speech, or any number of additional layers). It adds a number of search features to Lucene.

With BlackLab, you can search for complex patterns of words (e.g. “find all nouns preceded by two or three adjectives”). It can accurately highlight matches (not just simple terms) in the original document or show them in a keyword-in-context (KWIC) view. It can quickly sort or group large result sets based on several criteria, including the exact words matched or words surrounding the match. It can also search inside specific XML tags, so you can search for people or places, for example.

BlackLab supports a number of input formats out of the box. Adding support for a new input format is a matter of writing a configuration file.

Who uses BlackLab?

These fine projects and people!

BlackLab vs. others

Other corpus engines exist and have many great features. Each has their own strengths and weaknesses.

What does BlackLab offer that others may not?

• prioritizes good documentation and ease of use
• allows sorting and grouping hits on different properties, including context
• works with different input data formats
• supports (dependency) relations search, and you can combine this with token-based queries
• supports parallel corpora
• supports capturing parts of matches
• can be customized in many ways using various types of plugins (add your own query language, or add functions to BCQL, add a conversion step to indexing, etc.)
• provides a REST API so it's easy to use from any programming languages

Project goals?

Roughly, these are our project goals, in no particular order:

• ease of use (continuously improve documentation, error messages, etc.)
• ease of development and maintenance (continuously improve code quality)
• versatility (add useful features)
• scalability (make it possible and convenient to build and publish huge corpora)
• inclusivity (be responsive, foster new users and contributors)

See this full article.

Do you use AI?

The topic of AI / machine learning can be controversial. Let's clarify if and how we use it in BlackLab.

Search

Search in BlackLab does not use AI. Most of its users are in the scientific community, so reproducible results are important. A user specifies what they want to search for, and BlackLab finds it exactly as specified.

"Search by description"

Because not everyone is familiar with the included query language (BCQL), we may develop an (optional) feature where you can input what you are looking for ("find the most used adjectives that start with a b") which is then converted into a query for you. The user sees the query and can check if it looks correct. That feature would use generative AI (trained on example queries written by us), but it could be disabled if desired. It would likely not be a core part of BlackLab but a separate service that works with BlackLab Frontend.

Tokenizing and tagging

Annotating input texts with linguistic information ("tagging") is outside the scope of BlackLab itself. There are various tools available that can be combined with BlackLab for this purpose. Some of these tools use (a form of) pattern-recognizing AI.

Development

Programmers working on BlackLab may use AI-based tools (such as GitHub Copilot, Claude Code or OpenAI Codex) to help them write code more efficiently, like most programmers nowadays. However, we believe that humans should always be fully in the loop and are committed to line-by-line reviewing anything written by an LLM.

We recognize there are problems with AI, but also think that it can greatly benefit open-source projects like ours when used carefully.

Is BlackLab fast?

We don't do regular search-offs, but in the past we've determined that BlackLab offered similar speeds to other corpus search engines. Each engine tends to be better at some types of queries than others.

We've done a lot of work over the years to improve BlackLab's performance, and we invite you to see for yourself.

Distributed search?

This is a feature we are looking in to, both to improve scalability and flexibility (maintaining a large corpus made up of various subcorpora).

See our future plans.

Future plans

We will keep improving BlackLab as time goes on.

Broadly speaking, these are our plans for the future:

• Improve performance and memory requirements (ongoing)
• Keep up with new Lucene versions (ongoing)
• Scale to larger corpora by adding support for distributed search, either through integration with Solr (experiment in progress) or via another route
• Keep up to date with new Java versions (ongoing)
• Make accessing and highlighting the original content of documents more flexible (stored in the index or accessed through an API)
• Enhance dependency relations search
• Integrate improvements and suggestions from the community (issues and pull requests welcome!)

For more (somewhat technical) details, see the corresponding BARKs (short descriptions of planned development and other information).