Collocations

In-development feature

Try this feature on dev (Docker image or branch). It will be finalized for version 5.0.

A collocation is a series of words or terms that co-occur more often than would be expected by chance. For example, bark and yap are collocates of the keyword dog.

Collocations in BlackLab can be proximity-based (i.e. words that occur within 5 positions of dog) or relation-based (i.e. words that have some relation with dog).

You can find collocations using a regular /hits request with the group parameter, but there is also a /collocations endpoint that makes it a bit more convenient. It takes the components that make up a collocation request and constructs the correct query, grouping property, and group scorer configuration.

URL

• /blacklab-server/<corpus-name>/collocations (API v4)
• /blacklab-server/corpora/<corpus-name>/collocations (future API v5)

Method : GET

Parameters

Only patt is required; other parameters are optional.

Parameter	Description
filter	Lucene Query Language document filter query, to find collocations in part of the corpus
patt	BlackLab Corpus Query Language (BCQL) collocation keyword: the pattern to find collocations for, e.g. [lemma="dog"] to find collocates of dog(s).
collpatt	Collocate filtering (BCQL) pattern, e.g. [pos="NOUN"] to find only noun collocates
colltype	type of collocations to find: proximity (default) or relsources/reltargets (for relation-based collocations; see below)
context	(proximity-based collocations) how close a word has to occur to be considered a collocate. Examples: 5 gives 5 words around the word, 5:10 gives 5 before and 10 after. Default: 5
within	(proximity-based collocations) whether a collocation needs to be contained within a specific tag. Example: s will require both parts to be within <s/>. Default: none
reltype	(relation-based collocations; optional) a specific relation type to match, or a regular expression to filter relation types.
group	Annotations to group on when gathering collocates (hit:<annotation>:<sensitivity>, like regular hits grouping). You can group on multiple annotations (comma-separated). Default: main annotation, i.e. hit:word:i.
scorertype	collocations found will be scored. This gives the hit group scorer to use. Builtin scorers are coll-dice (default) and coll-salience. See below.

In addition to these basic parameters, any parameter that can be added to a regular grouped hits request can be used (e.g. first, number, etc.). See here.

To view a collocate's hits, add the viewgroup parameter, containing that collocate group's identity value. The sort, first and number will apply to the hits within the collocate group. Keep context and other collocation parameters the same to preserve the group membership.

Scorers

Collocation groups are scored using a scorer formula:

name	description	formula
coll-dice	calculates the Dice Coefficient	(2 * f / (double) (f1 + f2))
coll-salience	calculates a log-based salience measure	log(f) * log(f * N / (f1 * f2)) / log(2)

In the above formulas, f is the frequency of the keyword and collocate occurring together; f1 and f2 are the frequencies of the words separately; N is the total size of the (sub)corpus selected by the document filter (or total cardinality of the relation type you searched for).

You can also use hit group scorers with a regular grouped /hits request. In this case, you should pass the scorer parameter with this JSON structure (commented for clarity):

{
  // id of the scorer to instantiate
  "id": "coll-dice",
  
  // document filter (if any; Lucene query language)
  "filter": "title:sea",
  
  // keyword BCQL pattern (used to determine keyword frequency)
  "patt": "\"boot\"",
  
  // relation type filter (if any, and if relation-based collocations) 
  "reltype": "nsubj",
  
  // what we grouped the hits on to determine our collocates and their frequencies
  "group": "hit:word:i",
  
  // use faster calculation that may still count terms in some deleted docs?
  // (defaults to true for better performance; experimental parameter, may be removed in the future)
  "loose": false,
}

You can implement a custom collocation scorer by writing a plugin of type HitGroupScorer. See plugins. Note that in the future, other types of hit group scorer might be added, but currently, collocation scorers are the only possible type.

Equivalent /hits requests

/collocations is just a convenience endpoint that performs a regular hits grouping. It exists because the /hits URL requires the user to understand the specific BCQL query needed, and the scorer configuration can be a bit cryptic.

Proximity-based

This proximity collocations request (shown without URL encoding for readability):

/CORPUSNAME/collocations?
    patt="boot"&
    collpatt=[pos="N.*"]&
    colltype=proximity&
    context=3:4&
    group=hit:word:i&
    scorertype=coll-dice

is exactly equivalent to this (shorter but more cryptic) grouped hits request:

/CORPUSNAME/hits?
    patt=meet([pos="N.*"],"boot",-3,4)&
    group=hit:word:i&
    scorer={"id":"coll-dice","patt":"\"boot\"","group":"hit:word:i"}

Relation-based

This proximity collocations request (shown without URL encoding for readability):

/CORPUSNAME/collocations?
    patt="boot"&
    collpatt=[pos="N.*"]&
    colltype=reltargets&
    reltype=nsubj&
    group=hit:word:i&
    scorertype=coll-dice

is exactly equivalent to this grouped hits request:

/CORPUSNAME/hits?
    patt=rspan("boot" -nsubj-> [pos="N.*"], "target")&
    group=hit:word:i&
    scorer={"id":"coll-dice","patt":"\"boot\"","group":"hit:word:i","reltype":"nsubj","colltype":"reltargets"}

Success Response

HTTP response code: 200 OK

The response will have the same structure as a /hits request with the group parameter. See here.