IForest spark scala

1
votes

I'm trying to use scala implementation of IForest https://github.com/titicaca/spark-iforest, but when i build (like reported in README with mvn clean package), it give me these errors:

[ERROR] IsolationForest_Spark/spark-iforest/src/main/scala/org/apache/spark/ml/iforest/IForest.scala:290: 
        error: value getAndSetParams is not a member of object org.apache.spark.ml.util.DefaultParamsReader
[ERROR]       DefaultParamsReader.getAndSetParams(model, metadata)
[ERROR]                           ^
[ERROR] IsolationForest_Spark/spark-iforest/src/main/scala/org/apache/spark/ml/iforest/IForest.scala:480:
        error: value create is not a member of object org.apache.spark.ml.util.Instrumentation
[ERROR]     val instr = Instrumentation.create(this, rddPerTree)

Does someone knows why? Thank you

scala version 2.11.12
spark version 2.4.0
maven version 3.5.2

I modified pom.xml adjusting version of scala, spark and maven, but I leave the same dependencies.

2
scalaapache-spark

2 Answers

2
votes

The project has been updated since this answer was written see F.Z.Yang's answer.

The spark iforest is using a spark internal api getAndSetParams which was moved in spark 2.4.0 according to this commit and issue SPARK-23455.

It seems spark iforest was compiled with spark 2.2.0, so you should probably start with that version.

You could try the pervious version (2.3.2) but if there are other breaking changes that may not work either.

1
votes

I have just upgraded spark-iforest to the latest spark version.

https://github.com/titicaca/spark-iforest

Have Fun ;)