Python NLTK tokenize sentence with wrong syntax from human errors

Question

I am looking for a way to handle sentence tokenizing task well.

I have this text extracted from a human written review for a restaurant

Nevertheless, the soup enhances the prawns well.In contrast, the fish offered is fresh and well prepared.

Note that, the period that is the boundary of first sentence is not separated by space. It is result from human error in writing. There are many sentences that were written like this that I can't ignore this one case.

So far I tried nltk sentence tokenizer in python but does not work as expected.

>>>import nltk.data
>>>tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
>>>sentences = tokenizer.tokenize(text)
>>>sentences
['Nevertheless, the soup enhances the prawns well.In contrast, the fish offered is fresh and well prepared.']

My expectation is it should be able to split the text into two sentences

['Nevertheless, the soup enhances the prawns well.', 'In contrast, the fish offered is fresh and well prepared.']

Any help is appreciated in advance

"does not work as expected" - what did you expect, and what happened instead? "does not tweak in my case" - what change did you make, and what happened? — jonrsharpe
@jonrsharpe i updated my question. I decided to remove the tweak link because it seems not relevant to the problem here. — sovanlandy
Clearly the tokenizer expects valid English text. Have you tried preprocessing to split the sentences, e.g. re.sub(r'(.([A-Z]))', r'. \2', text)? — jonrsharpe
What if there's a period that is not intended to be a sentence stop? Like The soup didn't taste well at all! Please contact me at [email protected] to get a detailed report. — ComputerFellow
@sovanlandy note that {1,} can be replaced with + and {1} left out entirely (see regex101.com/r/pU4jK7/1). — jonrsharpe

sovanlandy sovanlandy · Accepted Answer · 2014-08-19T07:39:08

I decided to use regex for preprocessing of the text. The regex i use was.

re.sub(r'(\w{2})([.!?]+)(\w+)', r'\1\2 \3', text)

It has 3 groups. Group 1 is before the punctuation (\w{2}). Group 2 is the punctuation which can be [!?.] and can repeat more than once so it is ([.!?]{1,}). Group 3 is the next word after punctuation which can anywhere be 1 or more character word like "I" (\w{1}) .

Python NLTK tokenize sentence with wrong syntax from human errors

1 Answers