Submit Spark Streaming job via DataProc UI

Question

How do I specify multiple jar files on the DataProc UI (I mean on the Web browser). For example, from command line I can start the job as:

export SPARK_MASTER=local[8]
export DEPENDENCIES=/home/xxx/.ivy2/cache/org.apache.bahir/spark-streaming-twitter_2.11/jars/spark-streaming-twitter_2.11-2.0.1.jar,/home/xxx/.ivy2/cache/org.twitter4j/twitter4j-core/jars/twitter4j-core-4.0.4.jar,/home/xxx/.ivy2/cache/org.twitter4j/twitter4j-stream/jars/twitter4j-stream-4.0.4.jar
/usr/bin/spark-submit \
      --master $SPARK_MASTER \
      --jars $DEPENDENCIES \
      --class me.baghino.spark.streaming.twitter.example.TwitterSentimentScore \
      target/scala-2.11/spark-twitter-stream-example_2.11-1.0.0.jar

I copied all these files to my bucket on Google Storage & then under Jar files I entered:

gs://mybucket/testdata/spark-twitter-stream-example_2.11-1.0.0.jar:gs://mybucket/testdata/spark-streaming-twitter_2.11-2.0.1.jar:gs://mybucket/testdata/twitter4j-core-4.0.4.jar:gs://mybucket/testdata/twitter4j-stream-4.0.4.jar

Also tried using commas:

gs://mybucket/testdata/spark-twitter-stream-example_2.11-1.0.0.jar,gs://mybucket/testdata/spark-streaming-twitter_2.11-2.0.1.jar,gs://mybucket/testdata/twitter4j-core-4.0.4.jar,gs://mybucket/testdata/twitter4j-stream-4.0.4.jar

I also tried adding --jars under Arguments. That didn't work either.

rsantiago rsantiago · Accepted Answer · 2018-09-04T17:42:55

Hoping this answer still helps. In the dashboard 'Submiting a job' in the field 'Jar files', you have to enter one file per entry. This means that you need to type the Enter key, just like the Arguments field behaves. I found in documentation this clue: "Press to open a new text box for each additional argument." This is an example for your jars:

Submit Spark Streaming job via DataProc UI

1 Answers