Sentence tokenization in spacy is bad (?)

Question

Why is the sentence splitter/tokenizer from spacy works bad ? nltk seems to work fine. Here's my little experience:

import spacy
nlp = spacy.load('fr')
import nltk

text_fr = u"Je suis parti a la boulangerie. J'ai achete trois croissants. C'etait super bon."


nltk.sent_tokenize(text_fr)
# [u'Je suis parti a la boulangerie.',
# u"J'ai achete trois croissants.",
# u"C'etait super bon."


doc = nlp(text_fr)
for s in doc.sents: print s
# Je suis parti
# a la boulangerie. J'ai
# achete trois croissants. C'
# etait super bon.

I notice the same behavior for english. For this piece of text:

text = u"I went to the library. I did not know what book to buy, but then the lady working there helped me. It was cool. I discovered a lot of new things."

I get with spacy (after nlp=spacy.load('en')):

I
went to the library. I
did not know what book to buy, but
then the lady working there helped me. It was cool. I discovered a
lot of new things.

vs this with nltk which looks good:

[u'I went to the library.',
 u'I did not know what book to buy, but then the lady working there helped me.',
 u'It was cool.',
 u'I discovered a lot of new things.']

from spacy.io/usage/processing-pipelines: "Currently, sentence segmentation is based on the dependency parse, which doesn't always produce ideal results" — dada
My spacy version was too old (0.100), with v2, spacy works as expected — dada
Have in mind that you can define a custom sentence splitter: spacy.io/usage/processing-pipelines#component-example1 — erickrf

dada dada · Accepted Answer · 2018-01-16T16:09:14

I don't now how, but it turned out that I was using an old version of spacy (v 0.100). I installed the latest spacy again (v2.0.4) and now the sentence splitting is more coherent

Sentence tokenization in spacy is bad (?)

1 Answers