load pre-trained word embeddings

Question

I want to load pre-trained word embeddings from google news

model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
print (model.wv.vocab)

But the error is showing:

UnicodeEncodeError: 'ascii' codec can't encode character '\u2022' in position 62425: ordinal not in range(128)

How do I fix this? as I want to list all the words in the word embeddings and do the average for the sentence embedding.

Keith Keith · Accepted Answer · 2018-02-01T00:19:28

I'm loading them the same way and don't have that problem - I suspect that it's the print statement. Probably your stdout is setup for ascii only, whether it's in jupyter or on a terminal. To avoid that problem, I'd suggest opening a file with encoding like

with open("vocab.txt", "w", encoding="utf8") as vocab_out:
    for word in model.wv.vocab:
        vocab_out.write(word + "\n")

load pre-trained word embeddings

1 Answers