Running Spark Job on Zeppelin

2
votes

I have written a custom spark library in scala. I am able to run this successfully as a spark-submit step by spawning the cluster and running the following commands. Here I first get my 2 jars by - 

aws s3 cp s3://jars/RedshiftJDBC42-1.2.10.1009.jar .
aws s3 cp s3://jars/CustomJar .

and then i run my spark job as 

spark-submit --deploy-mode client --jars RedshiftJDBC42-1.2.10.1009.jar --packages com.databricks:spark-redshift_2.11:3.0.0-preview1,com.databricks:spark-avro_2.11:3.2.0 --class com.activities.CustomObject CustomJar.jar

This runs my CustomObject successfully. I want to run the similar thing in Zeppelin, But I do not know how to add jars and then run a spark-submit step?

1
apache-sparkapache-zeppelin

1 Answers

2
votes

You can add these dependencies to the Spark interpreter within Zeppelin:

  • Go to "Interpreter"
  • Choose edit and add the jar file
  • Restart the interpreter

More info here

EDIT You might also want to use the %dep paragraph in order to access the zvariable (which is an implicit Zeppeling context) in order to do something like this:

%dep
z.load("/some_absolute_path/myjar.jar")