SciKit-Learn: Trouble with TfidfVectorizer

Question

I'm trying to use TFIDF to get features from titles of text articles. I'm doing the following:

from sklearn.feature_extraction.text import TfidfVectorizer
corpus_title = result_df['_title'].tolist()
tfidf_transformer_title = TfidfVectorizer(min_df = 1, ngram_range = (1,1), use_idf = True, stop_words='english')
tfidf_df_title = tfidf_transformer_title.fit_transform(corpus_title)
tfidf_df_title

However, I get an error at this line:

----> 4 tfidf_df_title = tfidf_transformer_title.fit_transform(corpus_title)

The error is:

    205 
    206         if self.lowercase:
--> 207             return lambda x: strip_accents(x.lower())
    208         else:
    209             return strip_accents

AttributeError: 'NoneType' object has no attribute 'lower'

I'm not sure how it's possible to get this error. I checked the docs and it looks like TfidfVectorizer uses UTF-8 as its default encoding.

Any idea how to fix?

Thanks!

@cᴏʟᴅsᴘᴇᴇᴅ is there a way to skip these entries? — anon_swe
Not familiar with pandas, but you could probably take a look at exactly what data is causing the problem, and then use boolean indexing to filter it out. — cs95
Using result_df['_title'].dropna() should drop what appear to be empty entries in your corpus — elz

Yousra Gad Yousra Gad · Accepted Answer · 2017-07-07T13:39:34

try this:

tfidf_transformer_title = TfidfVectorizer(min_df = 1,lowercase = False, ngram_range = (1,1), use_idf = True, stop_words='english')

SciKit-Learn: Trouble with TfidfVectorizer

1 Answers