Is there an official way to support both Spark 1.6.2 and 2.0.0 on Hadoop yarn 2.7.2 cluster?

Question

I have a cluster running Hadoop yarn 2.7.2 with dynamic allocation enabled for Spark 1.6.2.

Is there an official way to support both Spark 1.6.2 and 2.0.0? Because when I tried to submit an application from Spark 2.0.0 client, exception happened in driver like below:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.network.util.JavaUtils.byteStringAs(Ljava/lang/String;Lorg/apache/spark/network/util/ByteUnit;)J
    at org.apache.spark.internal.config.ConfigHelpers$.byteFromString(ConfigBuilder.scala:63)
    at org.apache.spark.internal.config.ConfigBuilder$$anonfun$bytesConf$1.apply(ConfigBuilder.scala:197)
    at org.apache.spark.internal.config.ConfigBuilder$$anonfun$bytesConf$1.apply(ConfigBuilder.scala:197)
    at org.apache.spark.internal.config.TypedConfigBuilder.createWithDefaultString(ConfigBuilder.scala:131)
    at org.apache.spark.internal.config.package$.<init>(package.scala:41)
    at org.apache.spark.internal.config.package$.<clinit>(package.scala)
    at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMaster.scala:69)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:785)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:71)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:70)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:70)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:784)
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

Navarro Navarro · Accepted Answer · 2016-10-10T09:21:12

This feature is supported by Hortonwork's HDP distribution, I have a cluster running HDP 2.5, which supports Hadoop 2.7.3, Spark 1.6.2 and 2.0.0 on Centos7.

I have not experienced any problem while using either Spark and Spark2 jobs.

How did you install and configure both Spark versions? You can give a try to HDP sandbox and use as inspiration how is Spark & Spark2 configured for your own cluster.

Is there an official way to support both Spark 1.6.2 and 2.0.0 on Hadoop yarn 2.7.2 cluster?

1 Answers