I am new to NLP. I want to use Named Entity Recognition algorithm for extract the names and locations from the text without using any libraries.
Example:
American Airlines said it would launch a direct flight to Bengaluru.
Answer:
Entity: American Airlines
Location: Bengaluru
What is the best practice of Named Entity? Is it like storing all the names and locations on CSV file and comparing with the sentence?
As Manjur said, nowadays the best option is Deep learning. The most common is BERT, Pre-training of Deep Bidirectional Transformers for Language Understanding and variants. These are variants of Transformers encoder, which is better for extracting information than the decoder, like GPTs.
These models use fine-tuning for tasks like NER. In fact, they are already finetuned versions for Localizations and Persons. You can find implementations in Spacy, for example. But if you want better accuracy and you have experience with pytorch or tensorflow, preprocessing corpora, and you have labelled data to use (They also exist datasets only with these entities), then you can fine tune it yourself.
Libraires: https://medium.com/@b.terryjack/nlp-pretrained-named-entity-recognition-7caa5cd28d7b#:~:text=
Different approaches and SOTA: https://primer.ai/blog/a-new-state-of-the-art-for-named-entity-recognition/
The whole implementation of NER with BERT using a CSV (It's from Kaggle so you can also download the data set): https://www.kaggle.com/abhishek/entity-extraction-model-using-bert-pytorch
You can implement Named Entity Recognition in many ways:
One can treat this problem as multi-class classification problem where named entities are our labels so we can apply different classification algorithms. The problem with this method is that we miss out the context of the word in the sentence. Identifying and labeling the words required thorough understanding of word in the sentence.
To build state-of-art NER, we need to dig into deep learning approaches. As we know the context of the word play an important role, considering that the text is a sequential data format Long Short Term Memory (LSTM) play an important role. Not any type of LSTM will give best result. We need to user Bi-Directional LSTM because standard LSTM make predictions using the past information in a sequence of the text. For NER, since the context covers past and future labels in a sequence, we need to take both the past and the future information into account. A Bi-Directional LSTM is a combination of two LSTMs — one runs forward from right to left and another runs backward from left to right.
