Yarn - executor for spark job

0
votes
Process spark = new SparkLauncher()
.setAppResource("myApp.jar")
.setMainClass("com.aa.bb.app")
.setMaster("yarn")
.setDeployMode( "cluster")
.addAppArgs( data)
.launch();

This is how I executed my spark jar to yarn cluster. Here are some question below.

  1. Is this processing with a executor? ( 1 spark-submit per 1 yarn executor?)
  2. How should I executed multi spark job concurrently? (Where should I set dynamic allocation(spark.dynamicAllocation.enabled)?)
  3. Where should I set number of executor configuration? In java code? In yarn xml?
  4. If I set number of executor as 2, and process single job, one of executor will do nothing?
1
hadoopapache-sparkhadoop-yarn
why do you use this method to start a spark application? you could have used spark-submit instead. <br/> another thing, I think you should first read spark documentation to understand the spar executor. Running a jar file with spark does not mean it is running on one executor or not. you should dive more in the code that uses spark, the data input size, if there will be map jobs which will starts an executor. Kindly, check spark documentation and refine your question spark.apache.org/docs/2.1.0/programming-guide.html spark.apache.org/docs/2.1.0/running-on-yarn.html - user1314742

1 Answers

0
votes

Need to do nothing for this. It allocated automatically.

enter image description here