Faster Lemmatization techniques in Python

Question

I am trying to find out a faster way to lemmatize words in a list (named text) using the NLTK Word Net Lemmatizer. Apparently this is the most time consuming step in my whole program(used cProfiler to find the same).

Following is the piece of code that I am trying to optimize for speed -

def lemmed(text):
    l = len(text)
    i = 0
    wnl = WordNetLemmatizer()
    while (i<l):
        text[i] = wnl.lemmatize(text[i])
        i = i + 1
    return text

Using the lemmatizer decreases my performance by 20x. Any help would be appreciated.

Decreases the performance of what by 20x? What do you need lemmatized forms for? — rmalouf
@rmalouf if I remove this function my program runs 20x faster. I need to pre-process data before running an algorithm on it. Hence the need for lemmatized forms. — Shivansh Singh
The reason I ask is that there are faster lemmatizers/stemmers than the wordnet one, but they also give different results. The answer is going the depend on exactly what your algorithm needs as input, and on how fast is going to be fast enough for your application. It's hard to know how to answer a question like this without knowing the details about what the target is. — rmalouf
@rmalouf I would love to know about the faster lemmatizers. My input is a list of words OCR'ed from a document and I am looking to classify that document based on the words. I know the labels so it would fall under supervised learning if that helps in any way. — Shivansh Singh

Alec Alec · Accepted Answer · 2016-06-24T19:11:05

If you have a few cores to spare, try using the multiprocessing library:

from nltk import WordNetLemmatizer
from multiprocessing import Pool

def lemmed(text, cores=6): # tweak cores as needed
    with Pool(processes=cores) as pool:
        wnl = WordNetLemmatizer()
        result = pool.map(wnl.lemmatize, text)
    return result


sample_text = ['tests', 'friends', 'hello'] * (10 ** 6)

lemmed_text = lemmed(sample_text)

assert len(sample_text) == len(lemmed_text) == (10 ** 6) * 3

print(lemmed_text[:3])
# => ['test', 'friend', 'hello']

Faster Lemmatization techniques in Python

1 Answers