Extract document-topic matrix from Pyspark LDA Model

Question

I have successfully trained an LDA model in spark, via the Python API:

from pyspark.mllib.clustering import LDA
model=LDA.train(corpus,k=10)

This works completely fine, but I now need the document-topic matrix for the LDA model, but as far as I can tell all I can get is the word-topic, using model.topicsMatrix().

Is there some way to get the document-topic matrix from the LDA model, and if not, is there an alternative method (other than implementing LDA from scratch) in Spark to run an LDA model that will give me the result I need?

EDIT:

After digging around a bit, I found the documentation for DistributedLDAModel in the Java api, which has a topicDistributions() that I think is just what I need here (but I'm 100% sure if the LDAModel in Pyspark is in fact a DistributedLDAModel under the hood...).

In any case, I am able to indirectly call this method like so, without any overt failures:

In [127]: model.call('topicDistributions')
Out[127]: MapPartitionsRDD[3156] at mapPartitions at PythonMLLibAPI.scala:1480

But if I actually look at the results, all I get are string telling me that the result is actually a Scala tuple (I think):

In [128]: model.call('topicDistributions').take(5)
Out[128]:
[{u'__class__': u'scala.Tuple2'},
 {u'__class__': u'scala.Tuple2'},
 {u'__class__': u'scala.Tuple2'},
 {u'__class__': u'scala.Tuple2'},
 {u'__class__': u'scala.Tuple2'}]

Maybe this is generally the right approach, but is there way to get the actual results?

I know that the LDA functionality in Spark is still in development, but it seems bizarre that there's no straightforward way of getting this info out of the model... — moustachio
I think there is an another issue here. As pointed to me by Jason Lenderman Spark LDA doesn't implement LSA but a variant of PLSI. It makes this matrices less useful directly. See also stackoverflow.com/a/32953813/1560062 — zero323
I see, but in that case a more or less equivalent solution would be to predict topics for the original training documents similar to the method described in the linked question, but as far as I can tell the necessary methods aren't implemented in the Python API. Are they hidden somewhere, or is there some other way of achieving this in Pyspark? — moustachio

moustachio moustachio · Accepted Answer · 2015-11-02T16:13:29

After extensive research, this is definitely not possible via the Python api on the current version of Spark (1.5.1). But in Scala, it's fairly straightforward (given an RDD documents on which to train):

import org.apache.spark.mllib.clustering.{LDA, DistributedLDAModel}

// first generate RDD of documents...

val numTopics = 10
val lda = new LDA().setK(numTopics).setMaxIterations(10)
val ldaModel = lda.run(documents)

# then convert to distributed LDA model
val distLDAModel = ldaModel.asInstanceOf[DistributedLDAModel]

Then getting the document topic distributions is as simple as:

distLDAModel.topicDistributions

Extract document-topic matrix from Pyspark LDA Model

3 Answers