Best scikit classifier for text classification task

Question

I am using scikit to do text classification of short phrases to their meaning. Some examples are:

"Yes" - label.yes
"Yeah" - label.yes
...
"I don't know" - label.i_don't_know
"I am not sure" - label.i_don't_know
"I have no idea" - label.i_don't_know

Everything worked pretty well using TfidfVectorizer and a MultinomialNB classifier.

The problem occurred when I added a new text/label pair:

"I" - label.i

Predicting the class for "I" still returns label.i_don't_know even though the text is exactly in the training data like this, which is probably due to the fact that the unigram "I" occurs more often in label.i_don't_know than in label.i.

Is there a classifier that will give comparable or better performance on this task and guarantee that predictions of training data elements are returned correctly?

This code illustrates the problem further:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

#instantiate classifier and vectorizer
clf=MultinomialNB(alpha=.01)
vectorizer =TfidfVectorizer(min_df=1,ngram_range=(1,2))

#Apply vectorizer to training data
traindata=['yes','yeah','i do not know','i am not sure','i have no idea','i'];
X_train=vectorizer.fit_transform(traindata)

#Label Ids
y_train=[0,0,1,1,1,2];

#Train classifier
clf.fit(X_train, y_train)

print clf.predict(vectorizer.transform(['i']))

The code outputs label 1, but the correct classification would be label 2.

Without a reproducible example of your code, it'll be very hard to say whether the solution would be to use a different classifier, or change the way you're using the current classifiers. — Marius

mbatchkarov mbatchkarov · Accepted Answer · 2013-07-02T12:57:22

The problem is not with the classifier, it is with the vectorizer. TfidfVectorizer has a parameter token_pattern : string, which is a "Regular expression denoting what constitutes a “token”, only used if tokenize == ‘word’. The default regexp select tokens of 2 or more letters characters (punctuation is completely ignored and always treated as a token separator)." (emphasis added). The tokenizer throws out the word i, resulting in an empty document. Naive Bayes then classifies that as class 1, because this is the most frequent class in the training data.

Depending on the data, you might want to consider using a uniform prior for Naive Bayes.

Further hints as to why things may not be working:

There might be some other oddity in the way your pipeline is set up. I find it useful to inspect the inputs and outputs of each stage (tokenizer, vectorizer, classifier, etc). Investing some time into writing a unit test will save you lots of time in the long run.

Once you are satisfied everything is working fine, try evaluating your classifier on the test data. My suspicion is that there is considerable overlap between your classes, particularly label.i_don't_know and label.i. If this is the case, the classifier will perform poorly.

Best scikit classifier for text classification task

1 Answers