How to concatenate word vectors to form sentence vector

3
votes

I have learned in some essays (Tomas Mikolov...) that a better way of forming the vector for a sentence is to concatenate the word-vector.

but due to my clumsy in mathematics, I am still not sure about the details.

for example,

supposing that the dimension of word vector is m; and that a sentence has n words.

what will be the correct result of concatenating operation?

is it a row vector of 1 x m*n ? or a matrix of m x n ?

1
machine-learningdeep-learningnlpword2vec
where did Mikolov mention that? A reference to the paper, etc is useful. To answer your question, concatenation is just putting vectors one after the other. So if you have 3 vectors of 1x100 size, the sentence vector will 1x300. – user3639557

1 Answers

5
votes

There are at least three common ways to combine embedding vectors; (a) summing, (b) summing & averaging or (c) concatenating. So in your case, with concatenating, that would give you a 1 x m*a vector, where a is the number of sentences. In the other cases, the vector length stays the same. See gensim.models.doc2vec.Doc2Vec, dm_concat and dm_mean - it allows you to use any of those three options [1,2].

[1] http://radimrehurek.com/gensim/models/doc2vec.html#gensim.models.doc2vec.LabeledLineSentence

[2] https://github.com/piskvorky/gensim/blob/develop/gensim/models/doc2vec.py