Improving DOC2VEC Gensim efficiency

Question

I am trying to train Gensim Doc2Vec model on tagged documents. I have around 4000000 documents. Following is my code:

import pandas as pd
import multiprocessing
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
from nltk.stem import WordNetLemmatizer
import logging
from tqdm import tqdm
from gensim.models import Doc2Vec
from gensim.models.doc2vec import TaggedDocument
import os
import re



def text_process(text):
    logging.basicConfig(format="%(levelname)s - %(asctime)s: %(message)s", datefmt='%H:%M:%S', level=logging.INFO)
    stop_words_lst = ['mm', 'machine', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', 'first', 'second', 'third', 'plurality', 'one', 'more', 'least', 'at', 'example', 'memory', 'exemplary', 'fourth', 'fifth', 'sixth','a', 'A', 'an', 'the', 'system', 'method', 'apparatus', 'computer', 'program', 'product', 'instruction', 'code', 'configure', 'operable', 'couple', 'comprise', 'comprising', 'includes', 'cm', 'processor', 'hardware']
    stop_words = set(stopwords.words('english'))

    temp_corpus =[]
    text = re.sub(r'\d+', '', text)
    for w in stop_words_lst:
        stop_words.add(w)
    tokenizer = RegexpTokenizer(r'\w+')
    word_tokens = tokenizer.tokenize(text)
    lemmatizer= WordNetLemmatizer()
    for w in word_tokens:
        w = lemmatizer.lemmatize(w)
        if w not in stop_words:
            temp_corpus.append(str(w))
    return temp_corpus

chunk_patent = pd.DataFrame()
chunksize = 10 ** 5
cores = multiprocessing.cpu_count()
directory = os.getcwd()
for root,dirs,files in os.walk(directory):
    for file in files:
       if file.startswith("patent_cpc -"):
           print(file)
           #f=open(file, 'r')
           #f.close()
           for chunk_patent_temp in pd.read_csv(file, chunksize=chunksize):
                #chunk_patent.sort_values(by=['cpc'], inplace=True)
                #chunk_patent_temp = chunk_patent_temp[chunk_patent_temp['cpc'] == "G06K7"]
                if chunk_patent.empty:
                    chunk_patent = chunk_patent_temp
                else:
                    chunk_patent = chunk_patent.append(chunk_patent_temp)
train_tagged = chunk_patent.apply(lambda r: TaggedDocument(words=text_process(r['text']), tags=[r.cpc]), axis=1)
print(train_tagged.values)

if os.path.exists("cpcpredict_doc2vec.model"):
    doc2vec_model = Doc2Vec.load("cpcpredict_doc2vec.model")
    doc2vec_model.build_vocab((x for x in tqdm(train_tagged.values)), update=True)
    doc2vec_model.train(train_tagged, total_examples=doc2vec_model.corpus_count, epochs=50)
    doc2vec_model.save("cpcpredict_doc2vec.model")
else:
    doc2vec_model = Doc2Vec(dm=0, vector_size=300, min_count=100, workers=cores-1)
    doc2vec_model.build_vocab((x for x in tqdm(train_tagged.values)))
    doc2vec_model.train(train_tagged, total_examples=doc2vec_model.corpus_count, epochs=50)
    doc2vec_model.save("cpcpredict_doc2vec.model")

I have tried modifying the Doc2vec parameters but without any luck.

On the same data I have trained Word2vec model, which is much accurate in comparison to the doc2vec model. Further, "most_similar" results for word2vec model is very different from the doc2vec model.

Following is the code for searching most similar results:

from gensim.models import Word2Vec
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer
from nltk.stem import WordNetLemmatizer
import logging
from gensim.models import Doc2Vec
import re

def text_process(text):
    logging.basicConfig(format="%(levelname)s - %(asctime)s: %(message)s", datefmt='%H:%M:%S', level=logging.INFO)
    stop_words_lst = ['mm', 'machine', '1', '2', '3', '4', '5', '6', '7', '8', '9', '0', 'first', 'second', 'third', 'example', 'memory', 'exemplary', 'fourth', 'fifth', 'sixth','a', 'A', 'an', 'the', 'system', 'method', 'apparatus', 'computer', 'program', 'product', 'instruction', 'code', 'configure', 'operable', 'couple', 'comprise', 'comprising', 'includes', 'cm', 'processor', 'hardware']
    stop_words = set(stopwords.words('english'))
    #for index, row in df.iterrows():
    temp_corpus =[]
    text = re.sub(r'\d+', '', text)
    for w in stop_words_lst:
        stop_words.add(w)
    tokenizer = RegexpTokenizer(r'\w+')
    word_tokens = tokenizer.tokenize(text)
    lemmatizer= WordNetLemmatizer()
    for w in word_tokens:
        w = lemmatizer.lemmatize(w)
        if w not in stop_words:
            temp_corpus.append(str(w))
    return temp_corpus

model = Word2Vec.load("cpc.model")
print(model.most_similar(positive=['barcode'], topn=30))

model1 = Doc2Vec.load("cpcpredict_doc2vec.model")

pred_tags = model1.most_similar('barcode',topn=10)
print(pred_tags)

Further, the output of the aforementioned is cited below:

[('indicium', 0.36468246579170227), ('symbology', 0.31725651025772095), ('G06K17', 0.29797130823135376), ('dataform', 0.29535001516342163), ('rogue', 0.29372256994247437), ('certification', 0.29178398847579956), ('reading', 0.27675414085388184), ('indicia', 0.27346929907798767), ('Contra', 0.2700084149837494), ('redemption', 0.26682156324386597)]

[('searched', 0.4693435728549957), ('automated', 0.4469209909439087), ('production', 0.4364866018295288), ('hardcopy', 0.42193126678466797), ('UWB', 0.4197841286659241), ('technique', 0.4149003326892853), ('authorized', 0.4134449362754822), ('issued', 0.4129987359046936), ('installing', 0.4093806743621826), ('thin', 0.4016669690608978)]

gojomo gojomo · Accepted Answer · 2020-06-14T07:34:14

The Doc2Vec mode you've chosen, dm=0 (aka plain "PV-DBOW"), does not train word-vectors at all. Word vectors will still be randomly-initialized, due to shared code-paths of the different models, but never trained and thus meaingless.

So the results of your most_similar(), using a word as the query, will be essentially random. (Using most_similar() on the model itself, rather than its .wv word-vectors or .docvecs doc-vectors, should also be generating a deprecation warning.)

If you need your Doc2Vec model to train word-vectors in addition to the doc-vectors, use either the dm=1 mode ("PV-DM") or dm=0, dbow_words=1 (adding optional interleaved skip-gram word training to plain DBOW training). In both cases, words will be trained very similarly to a Word2Vec model (of the 'CBOW' or 'skip-gram' modes, respectively) – so your word-based most_similar() results should then be very comparable.

Separately:

if you have enough data to train 300-dimensional vectors, & discard all words with fewer than 100 occurrences, then 50 training epochs may be more than needed.
those most_similar() results don't particularly look like they're result of any lemmatization, as seems intended by your text_process() method, but maybe that's not an issue, or some other issue entirely. Note, though, that with sufficient data, lemmatization may be a superfluous step - all variants of the same word tend to wind up usefully near each other, when there are plenty of varied examples of al the word variants in real contexts.

Improving DOC2VEC Gensim efficiency

1 Answers