Column features must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually org.apache.spark.mllib.linalg.VectorUDT@f71b0bce

1
votes

spark version is 2.2.0 and scala version is 2.11。When I use ml lib, error occurs : " Column features must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually org.apache.spark.mllib.linalg.VectorUDT@f71b0bce."

This is my code:

import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions._

 val trainingData = dataSet
  .select(col("features"), col("label")).cache()

val lr = new LogisticRegression()
  .setMaxIter(maxIter)
  .setRegParam(regParam)
  .setElasticNetParam(0)
  .setThreshold(threshold)
  .setFitIntercept(false)

val lrModel = lr.fit(trainingData)

It confused me several days。Who can help me?

1
apache-spark
If you have a new question, please ask it as such, not completely rewrite existing one. - Alper t. Turker

1 Answers

3
votes

The error message is pretty clear you are using org.apache.spark.mllib.linalg.VectorUDT (the old MLlib API) while any new API (ML) requires org.apache.spark.ml.linalg.Vector.

You omitted the part of the code where you create dataSet, but you should replace:

org.apache.spark.mllib.linalg._

imports with:

org.apache.spark.ml._

and adjust upstream code accordingly.

Related: