Saving a Pipeline with DecisionTreeModel Spark ML

Question

Context:

I have a Spark ML pipeline that contains a VectorAssembler, StringIndexer, and a DecisionTreeClassifier. Using this pipeline I am able to successfully fit the model and transform my data frame. I would like to store this model for future use, but I keep getting the following error:

Pipeline write will fail on this Pipeline because it contains a stage which does not implement Writable. 
Non-Writable stage: dtc_9c04161ed2d1 of type class org.apache.spark.ml.classification.DecisionTreeClassificationModel

What I have tried:

val pipeline = new Pipeline().setStages(Array(assembler, labelIndexer, dt))
val model = pipeline.fit(dfIndexed)
model.write.overwrite().save("test/model/pipeline")

This works properly when I remove the classifier (i.e. dt). Is there a way of saving a DecisionTreeClassifier model?

My data consists of some indexed categorical values that I must map back to their original form (I know this will require using IndexToString). I am using Spark 1.6.

This deserves a JIRA with a feature requests (if it doesn't exist already). You can always use mllib model which is writable and pass data back but I doubt it will be satisfactory solution. — zero323
It's weird, because almost all the models got the save method — Alberto Bonsanto

gstvolvr gstvolvr · Accepted Answer · 2016-02-13T18:05:51

1

votes

This cannot be done as of Spark 1.6. The issue is being tracked here.

Saving a Pipeline with DecisionTreeModel Spark ML

1 Answers