SpaCy: Person entities missing

Question

I have some issues with spaCy missing Person entities. Below is an example when it seems like SpaCy is going blind when the name is preceded with certain titles. When you remove Labour MP (commented line) it recognises the name otherwise returns an empty list.

import spacy
from spacy.en import English
nlp_toolkit = English()
nlp = spacy.load('en')

text = u"In a recent tweet, Labour MP Luciana Berger sought clarification..."
#text = u"In a recent tweet, Luciana Berger sought clarification..."

all_tags = nlp(text)

person_list=[]
for ent in all_tags.ents:
    if ent.label_=="PERSON":
        person_list.append(str(ent))      
print person_list

Now, I'm using 1.8.2 version of SpaCy. But before I upgrade I would really like to know if this issue was fixed in the version 2 as it seems the upgrade is not that straight forward. Could anyone test it or suggest a workaround? Thanks!

UPDATE:

It gets even more interesting. If you change the name to a different more common name, it works.

text = u"In a recent tweet, Labour MP James Mill sought clarification..."
out: ['James Mill']

So it is to do with the name and not MP before it... Anyway - if anyone can check if it works properly with v2 I'll appreciate it!

Gunjan Gunjan · Accepted Answer · 2018-03-29T10:05:21

I tried your code with Spacy Version 2.0.7 and it does give "Luciana Berger" as an entity for the sentence

I am getting this list ['Luciana Berger'] for sentence "In a recent tweet, Labour MP Luciana Berger sought clarification..."

Also for other sentence, it gives "James Mill"

Maybe if you want to try the newer version, try to install in Virtual Environment, that way you can experiment on both versions. You can take help of this how to use virtual enviornment. In fact I will recommend using Virtual env as switching between versions is quite lengthy process. So better try before you switch.

Also just for reference, NERs are working in Spacy on "Training of Models", here is the link. So it can happen that not every name will be covered in NER.

SpaCy: Person entities missing

1 Answers