Word2Vec : Apache Spark and Tensorflow implementations

1
votes

Reading https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala this implementation of Word2Vec is a port of Google Word2Vec https://code.google.com/archive/p/word2vec/

Is this an implementation of paper 'Efficient Estimation of Word Representations in Vector Space' : https://arxiv.org/abs/1301.3781 ?

Tensorflow Word2Vec does reference paper 'Efficient Estimation of Word Representations in Vector Space' .

What then is difference between implementations of Apache Spark and Tensorflow Word2Vec and under what conditions should each be used ?

1
apache-sparktensorflowapache-spark-mllibword2vec

1 Answers

1
votes
  1. There are different ways to implement word2vec, but according to pyspark they do it by skip grams ( https://spark.apache.org/docs/2.1.0/mllib-feature-extraction.html#word2vec ). Tensorflow docs say they also use the skip gram model (https://www.tensorflow.org/tutorials/word2vec). From just glancing at the two docs, its seems like they calculate them the same way as well.
  2. Spark does really well on a distributed environment, and from what I am not aware of the benchmarks of tensorflow vs mllib as data gets bigger exactly, as Tensorflow distributed is fairly new.