spaCy: optimizing tokenization

0
votes

I'm currently trying to tokenize a text file where each line is the body text of a tweet:

"According to data reported to FINRA, short volume percent for $SALT clocked in at 39.19% on 12-29-17 http://www.volumebot.com/?s=SALT"
"@Good2go @krueb The chart I posted definitely supports ng going lower.  Gobstopper' 2.12, might even be conservative."
"@Crypt0Fortune Its not dumping as bad as it used to...."
"$XVG.X LOL. Someone just triggered a cascade of stop-loss orders and scooped up morons' coins. Oldest trick in the stock trader's book."

The file is 59,397 lines long (a day's worth of data) and I'm using spaCy for pre-processing/tokenization. It's currently taking me around 8.5 minutes and I was wondering if there were any way of optimising the following code to be quicker as 8.5 minutes seems awfully long for this process:

def token_loop(path):
    store = []
    files = [f for f in listdir(path) if isfile(join(path, f))]

    start_time = time.monotonic()
    for filename in files:
        with open("./data/"+filename) as f:
            for line in f:
                tokens = nlp(line.lower())
                tokens = [token.lemma_ for token in tokens if not token.orth_.isspace() and token.is_alpha and not token.is_stop and len(token.orth_) != 1]
                store.append(tokens)

    end_time = time.monotonic()
    print("Time taken to tokenize:",timedelta(seconds=end_time - start_time))

    return store

Although it says files, it's currently only looping over 1 file.

Just to note, I only need this to tokenize the content; I don't need any extra tagging etc.

2
machine-learningnlpspacy

2 Answers

2
votes

It sounds like you haven't optimised the pipeline yet. You'll get a significant speed up from disabling the pipeline components you don't need, like so:

nlp = spacy.load('en', disable=['parser', 'tagger', 'ner'])

This should get you down to about the two-minute mark, or better, on its own.

If you need a further speed up, you can look at multi-threading using nlp.pipe. Docs for multi-threading are here: https://spacy.io/usage/processing-pipelines#section-multithreading

0
votes

You can use nlp.pipe(all_lines) instead of nlp(line) for a faster processing

see Spacy's documentation - https://spacy.io/usage/processing-pipelines