How to get the probabilities of classes in Spark Naive Bayes classifier?

Question

I'm training a NaiveBayesModel in Spark, however when I'm using it to predict a new instance I need to get the probabilities for each class. I looked at the code of predict function in NaiveBayesModel and come up with the following code:

val thetaMatrix = new DenseMatrix (model.labels.length,model.theta(0).length,model.theta.flatten,true)
val piVector = new DenseVector(model.pi)
//val prob = thetaMatrix.multiply(test.features)

val x = test.map {p =>       
  val prob = thetaMatrix.multiply(p.features)          
  BLAS.axpy(1.0, piVector, prob)
  prob
}

Does this work properly? The line BLAS.axpy(1.0, piVector, prob) keeps giving me an error that the value 'axpy' is not found.

or is there any way that I can calculate the probabilities using the pi and theta — HHH

Francois G Francois G · Accepted Answer · 2015-08-06T10:00:35

In a recent pull-request this was added to the Spark trunk and will be released in Spark 1.5 (closing SPARK-4362). you can therefore call

def predictProbabilities(testData: RDD[Vector]): RDD[Vector]

or

def predictProbabilities(testData: Vector): Vector

How to get the probabilities of classes in Spark Naive Bayes classifier?

1 Answers