Get an error when using Livy to submit Spark jobs: User did not initialize spark context

0
votes

I am very new to Spark and I am following this document to submit Spark jobs via Livy https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-livy-rest-interface

Here's my command:

curl -k --user "username:password!" -v -H "Content-Type: application/json" -X POST -d '{ "file":"/test4spark/test4sparkhaha.jar", "className":"helloworld4spark.test" }' "https://xxx.azurehdinsight.net/livy/batches" -H "X-Requested-By: username"

The file test4sparkhaha.jar is a super simple Java application, it contains only one class, and there's only a main method that print "hahaha", nothing else... I exported the project in Eclipse to a Runnable Jar and I tried run it on my spark cluster using Java -jar and Spark-submit. Both worked good. Then I started to try submit the job via Livy and it always failed, I found below errors in Yarn logs:

19/11/06 14:36:06 ERROR ApplicationMaster: Uncaught exception: java.lang.IllegalStateException: User did not initialize spark context! at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:510) at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

Here's the Spark-submit command I tried:

spark-submit --class helloworld4spark.test test4sparkhaha.jar

It works perfectly...

Can you guys please help me to understand why it gives the error while Spark-submit works fine?

1
javaapache-sparkhadoopazure-hdinsight

1 Answers

1
votes

I guess you're trying to submit local .jar file with Livy. For spark-submit it works (submitting job to Yarn supports it), but for Livy server it doesn't.

To make it work you need to upload your jar to the hdfs/wasbs/adls/http accessible location (be sure that your Spark is configured to access that location).

Please refer the first paragraph of this guide.