Yarn keeps on killing Spark Application master on EMR

Question

I'm running a spark application on EMR emr-4.3.0 with 1 Master and 4 Nodes

Each one of them has 5GB memory and 2 cores.

The following are my spark-submit options

--class com.mobi.vserv.driver.Query5kPids1
--num-executors 4
--executor-memory 4g
--executor-cores 2
--driver-memory 4g

But I keep getting the following error

ERROR executor.CoarseGrainedExecutorBackend: Driver 10.225.19.144:56334 disassociated! Shutting down.

And finally Yarn Kills the Application master

ERROR ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM

1) Can I further Improve my spark-submit options for num-executors and executor-cores.

2) I have seen in the Perfomance Graph only 2 nodes CPU's utilization is above 50 % , while the other 2 has less than 5 %.

3) Interestingly , I have ran the same application on 2 nodes and 1 master with the same configurations given in Spark-submit and the application ran successfully. So whats the reason of this behaviour with 4 nodes

Answer 1

When running spark with yarn, it's never a good thing to manually force number of executor instance, use spark.dynamicAllocation.enabled=true instead.

This way spark will ask yarn politely, if it can have more resources to run, if yarn has, it will grant it.

To find the root cause of your scenario you need to see how much memory yarn has on yarn's application ui (default port 8088 on namenode). Since you are specifying yourself the memory for each executor, yarn can only allocate instance of that size, and also, yarn has to reserve a set of vCores and Memory reserve for next task.

You can use "yarn -applicationId app_name_id", to find application log on yarn, if it has an issue.