Doc2Vec Pre training and Inferring vectors

Question

Suppose I have trained the doc2vec model with 50000 documents and I want to infer vectors for a separate dataset containing 36000 documents. In this case will the inferred vectors be effective for the downstream task of classification, becasue my assumption is that the inferred vectors depend on the size of documents with which the model is trained.

Note: Both dataset i.e one used for training doc2vec and other for inferring vectors are unique but from the same domain of US supreme court.

Please correct me if I am wrong with valid reason.

Answer 1

With such a tiny dataset, no answer I can give will be as useful as just trying it to see if it works.

50000 is smallish for a training set, but some useful Doc2Vec results have been based on similar corpuses.

Vector inference, like training, reduces documents of any length to a fixed-size vector. (But note: gensim silently limits any text fed to a 2Vec model to 10000 tokens.)

But, if you've trained a model on documents that are all about 1000 words, then try inference on 10-word fragments, those doc-vectors might not be as useful, or useful in the same way, as inferred vectors on documents more similar to the training set. But you'd still need to try it to find out. (Also note: words not learned during training are completely ignored during inference, so later inferences on docs with many/all unknown words will be weak or meaningless.)

Is that the the case with your inference docs – they are very different from training docs in size & vocabulary? And if so, why? (Can you train with more representative documents?)

If the set of 36000 documents is fixed before training begins, it may also be valid/defensible to include them in the unsupervised Doc2Vec training. They're data, they help learn the domain lingo, and they don't have in them any form of the "right" answers for classification.