What is the entry point of Spark container in YARN cluster mode?

1
votes

What is the main entry point of a Spark executor when a Spark job is being run in Yarn cluster mode (for Spark 1.2.0+)?

What I am looking for is the Scala class name for the entry point of an Executor (which will be the process executing one of the tasks on a slave machine).

2
apache-sparkhadoop-yarn

2 Answers

2
votes
spark-submit --class [FULLY QUALIFIED CLASS NAME]
             --master yarn-cluster
             [JAR_TO_USE]

So, given the above, the class to be used is the one specified, which is loaded from the given jar, and it searches within that class for a static main method.

From SparkSubmit.scala:

val mainMethod = mainClass.getMethod("main", new Array[String](0).getClass)
2
votes

I think what you're asking about is org.apache.spark.executor.Executor or perhaps org.apache.spark.executor.Executor$TaskRunner. It is TaskRunner that will ultimately run a task.

It is regardless of the deploy mode (client vs cluster) or the cluster manager used, i.e. Hadoop YARN or Spark Standalone or Apache Mesos.