Running Sparkling-Water with external H2O backend

0
votes

I was following the steps for running Sparkling water with external backend from here. I am using spark 1.4.1, sparkling-water-1.4.16, I've build the extended h2o jar and exported the H2O_ORIGINAL_JAR and H2O_EXTENDED_JAR system variables. I start the h2o backend with 

java -jar $H2O_EXTENDED_JAR -md5skip -name test

But when I start sparkling water via 

./bin/sparkling-shell

and in it try to get the H2OConf with 

import org.apache.spark.h2o._
val conf = new H2OConf(sc).setExternalClusterMode().useManualClusterStart().setCloudName("test”)
val hc = H2OContext.getOrCreate(sc, conf)

it fails on the second line with 

<console>:24: error: trait H2OConf is abstract; cannot be instantiated
   val conf = new H2OConf(sc).setExternalClusterMode().useManualClusterStart().setCloudName("test")
              ^

I've tried adding the newly build extended h2o jar with --jars parameter either to sparkling water or standalone spark with no progress. Does any one have any hints?

2
scalaapache-sparkh2o
The tutorial is only few minor versions old, at the time of sparkling-water-1.4.16 the class was still abstract and therefore unusable in this way. – JaKu
External cluster is now supported only in 2.+ SW version (to some extend it will be available in SW1.6). Any reasons for using 1.4 version of Spark - it is really old (with respect to current state of Spark ecosystem). – Michal
Legacy system is the reason. We are planning upgrade in the near future, but we don't have enough time right now. – JaKu
Right now, we are not planning to back port the change into older Spark releases. However, technically it is not so complicated. – Michal

2 Answers

0
votes

This is unsupported for versions of Spark earlier than 2.0.

0
votes

Download the latest version of sparkling jar and add it to while starting the spark-shell:

./bin/sparkling-shell --master yarn-client --jars "<path to the jar located>"

Then run the code by setting the extended h2o driver:

import org.apache.spark.h2o._

val conf = new H2OConf(spark).setExternalClusterMode().useAutoClusterStart().setH2ODriverPath("//home//xyz//sparkling-water-2.2.5/bin//h2odriver-sw2.2.5-hdp2.6-extended.jar").setNumOfExternalH2ONodes(2).setMapperXmx("6G")

val hc = H2OContext.getOrCreate(spark, conf)