Python named entity recognition (NER): Replace named entities with labels

Question

I'm new to Python NER and am trying to replace named entities in text input with their labels.

from nerd import ner
input_text = """Stack Overflow is a question and answer site for professional and enthusiast programmers. It is a privately held website, the flagship site of the Stack Exchange Network,[5][6][7] created in 2008 by Jeff Atwood and Joel Spolsky."""
doc = ner.name(input_text, language='en_core_web_sm')
text_label = [(X.text, X.label_) for X in doc]
print(text_label)

The output is: [('2008', 'DATE'), ('Jeff Atwood', 'PERSON'), ('Joel Spolsky', 'PERSON')]

I can then extract the people, for example:

people = [i for i,label in text_label if 'PERSON' in label] 
print(people)

to get ['Jeff Atwood', 'Joel Spolsky'].

My question is how can I replace identified named entities in the original input text so that the result is:

Stack Overflow is a question and answer site for professional and enthusiast programmers. It is a privately held website, the flagship site of the Stack Exchange Network,[5][6][7] created in DATE by PERSON and PERSON.

Thanks so much!

Answer 1

You can loop over text_label and replace each text with the corresponding label

for text, label in text_label:
    input_text = input_text.replace(text, label)

print(input_text)

Answer 2

You may indeed loop over text and labels as @taha explained, but this is a bad idea in the general case! This loop may mix entities which have the same name but different types (or sometimes not be an entity) in the text, as you only rely on the label of the entity.

Consider for instance the following:

In 2000 I sent 2000 emails.

I saw a statue of Washington in Washington.

You won't be able to distinguish occurrences of "2000" or "Washington"! This may look like a rare case, but wouldn't it be better to avoid such errors, especially for very long documents?

As far as I understood, the ner python module looks like a simple binding to Spacy so I guess you can access the "start_char" and "end_char" values of each entity to avoid this, with some basic Python programming. By the way I also think this should be more efficient from a computational point of view.