I want to recognize named entities in a specific field (e.g. baseball). I know there are tools available like StanfordNER, LingPipe, AlchemyAPI and I have done a little testing with them. But what I want them to be is field specific as I mentioned earlier. How this is possible?
3
votes
By 'specific field', do you mean specific domain or area? Training the NER to a particular corpus for a specific domain may be one solution.
- Kenston Choi
@Kenston my mistake. yes I mean focused on a specific domain. If I train the NER classifier on names of baseball players for instance, can it keep on accepting only names of that domain and NOT the names of politicians or any other? Do they have such a behavior that I want?
- samsamara
I think it depends on the features used. If the features are more on cases (title or upper cases), then chances are the politician names would be included. Is having a gazetteer (list of player names) not ideal for you?
- Kenston Choi
But how do you create such a list containing names of all the players?
- samsamara
You can mine them from various sources in the Internet, like Wikipedia (en.wikipedia.org/wiki/List_of_Major_League_Baseball_players) or sports site. It depends on how exhaustive you want, and how difficult will be your test data. Consider that a baseball player was a former politician. Does the context show that a certain name is likely a player? And in what context do you want to determine the players' names? Or are you trying to determine if a certain name is likely a baseball player, meaning it has something to do with the name regardless of its context?
- Kenston Choi
2 Answers
3
votes
One approach may be to
Use a general (non-domain specific) tool to detect people's names
Use a subject classifier to filter out texts that are not in the domain
If the total size of the data set is sufficient and the accuracy of the extractor and classifier good enough, you can use the result to obtain a list of people's names that are closely related to the domain in question (e.g. by restricting the results to those that are mentioned significantly more often in domain-specific texts than in other texts).
In the case of baseball, this should be a fairly good way of getting a list of people related to baseball. It would, however, not be a good way to obtain a list of baseball players only. For the latter it would be necessary to analyse the precise context in which the names are mentioned and the things said about them; but perhaps that is not required.
Edit: By subject classifier I mean the same as what other people might refer to simply as categorization, document classification, domain classification, or similar. Examples of ready-to-use tools include the classifier in Python-NLTK (see here for examples) and the one in LingPipe (see here).
