Spark setAppName doesn't appear in Hadoop running applications UI

5
votes

I am running a spark streaming job and when I set the app name (a better readable string) for my spark streaming job, It doesn't appear in the Hadoop running applications UI. I always see the class name as the name in Hadoop UI

val sparkConf = new SparkConf().setAppName("BetterName")

How to set the job name in Spark, so it appears in this Hadoop UI ?

Hadoop URL for running applications is - http://localhost:8088/cluster/apps/RUNNING

enter image description here

[update] Looks like this is the issue only with Spark Streaming jobs, couldn't find solution on how to fix it though.

1
hadoopapache-spark

1 Answers

19
votes

When submitting a job via spark-submit, the SparkContext created can't set the name of the app, as the YARN is already configured for job before Spark. For the app name to appear in the Hadoop running jobs UI, you have to set it in the command line for spark-submit "--name BetterName". I kick off my job with a shell script that calls spark-submit, so added the name to the command in my shell script.