Why is TfidfVectorizer in scikit-learn showing this behavior?

Question

While creating TfidfVectorizer object if I pass explicitly even the default value for token_pattern arguement it throws error when I do fit_transform. Following is the error:

ValueError: empty vocabulary; perhaps the documents only contain stop words

I am doing this because eventually I want to pass a different value for token_pattern argument so that I can have single letter tokens also as part of my tfidf matrix.

Following is an example:

from sklearn.feature_extraction.text import TfidfVectorizer

train_set = ("The sky is blue.", "The sun is bright.")
vectorizer1 = TfidfVectorizer(ngram_range=(1, 2), max_df=1.0, min_df=1)
train_set_tfidf = vectorizer1.fit_transform(train_set)     #works fine
vectorizer2 = TfidfVectorizer(token_pattern=u'(?u)\b\w\w+\b', ngram_range=(1, 2), max_df=1.0, min_df=1)
train_set_tfidf = vectorizer2.fit_transform(train_set)     #throws error

Answer 1

It is advisable to prepend regular expressions with r, this should work:

vectorizer2 = TfidfVectorizer(token_pattern=r'(?u)\b\w\w+\b', ngram_range=(1, 2), max_df=1.0, min_df=1)
train_set_tfidf = vectorizer2.fit_transform(train_set)

This is a known bug in the documentation, but if you look at the source code they do use raw literals.