Using Conditional Random Fields for Nested named entity recognition

Question

My question is the following.

When we work on Named entity recognition tasks, in most cases the classic LSTM-CRF architecture is used, where the CRF uses the Viterbi decoder and the transition matrix to find the best tag sequence associated to a sentence.

My question is, if a token is now associated to multiple entities and not just one (which is the case of Nested NER), as in the case of Bank of China, where China is a location and Bank of China is an organization. Can the CRF algorithm be adapted for this case? That is, finding more than one possible path in the sequence.

Answer 1

This issue is related to the datasets format more than the LSTM-CRF in itself, i.e. you may indeed implement a LSTM-CRF that would recognize nested entities, without depth limitation, but they are rather rare.

Most of the machine learning (including LSTM-CRF) software are learned with a CoNLL (tab separated) dataset format, which is not convenient for unlimited depth nesting. Many dataset and systems implement a fixed depth nesting, using additional columns (more or less one per nesting depth). Software may use separate or joint learning for each depth or use cascading models.