Hortonworks Oozie Spark Action

0
votes

Has anyone managed to successfully run a Spark action through Oozie in HDP?

I am running on HDP 2.3.2 with Oozie 4.2.0. The spark action is set to run in yarn-cluster mode. The spark job is a dummy one, with no input file.

I have first hit the error over here and managed to bypass it by removing the following from the hdfs oozie spark lib folder (in my case: /user/oozie/share/lib/lib_20151116201309/spark/):

  • tachyon-0.5.0.jar
  • tachyon-client-0.5.0.jar (both because of this error)
  • spark-core_2.10-1.1.0.jar
  • spark-graphx_2.10-1.1.0.jar
  • spark-streaming_2.10-1.1.0.jar

and added the following in the workflow lib folder:

  • datanucleus-api-jdo-3.2.6.jar
  • datanucleus-core-3.2.10.jar
  • datanucleus-rdbms-3.2.9.jar
  • spark-1.4.1.2.3.2.0-2950-yarn-shuffle.jar
  • spark-assembly-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar
  • spark-examples-1.4.1.2.3.2.0-2950-hadoop2.7.1.2.3.2.0-2950.jar

Now I land on the following error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Call From hdp4/10.10.1.84 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused java.net.ConnectException: Call From hdp4/10.10.1.84 to 0.0.0.0:8032 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

I mention that:

  • both in job.properties file and in the yarn config file, the job tracker/resource manager port is set to 8032
  • the application runned for ~20min (I ran it multiple times and observed this timing)
1
apache-sparkooziehortonworks-data-platform
Can you show us what you have tried - Peter Smith
What version of HDP are you using? What have you tried, and what errors are you running into? - Ton Torres
@PeterSmith I have added more info, thank you - Murukan
@TonTorres I have added more info, thank you - Murukan

1 Answers

0
votes

Here are the steps that I performed to get SparkAction to work on HDP 2.3.4.

  1. Backup the Oozie sharelib spark folder and delete all the jars inside except oozie-sharelib-<spark-version>.jar. In my case I simply renamed the spark folder to spark-backup and created a new spark folder, then copied the jar I mentioned.
  2. Copy all of the jars from $SPARK_HOME/lib to the Oozie sharelib spark folder.
  3. Restart Oozie
  4. Changed yarn.resourcemanager.address to <myhost>:8032 via Ambari
  5. Restarted YARN

I had more or less the same problems before, so I posted this question on the Hortonworks Community, which is what helped me to finally get it working.