EMR configuration for spark.dynamicAllocation doesn't match Spark official documentation

Question

According to official Spark documentation (http://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup), when using "spark.dynamicAllocation" option with YARN, you need to:

In the yarn-site.xml on each node, add spark_shuffle to yarn.nodemanager.aux-services ...

set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService

Despite that AWS EMR documentation says, that

"..Spark Shuffle Service is automatically configured by EMR. (http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-configure.html)

I've noticed, that "yarn.nodemanager.aux-services" in "yarn-site" on EMR nodes is set to:

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,</value>
</property>

and no section for "yarn.nodemanager.aux-services.spark_shuffle.class" is added at all.

I'm a bit new to Spark/Hadoop ecosystem, so this raised a few questions in my head:

Why I'm still able to successfully run Spark jobs with "spark.dynamicAllocation" set to "true", while the basic configuration requirements are not met? Does this mean that Spark somehow could use "mapreduce_shuffle" as a fallback?
If the assumption above (Spark falls back to "mapreduce_shuffle") is true, are there possible performance (other?) implications from using improper shuffle class ("mapreduce_shuffle" maps to "org.apache.hadoop.mapred.ShuffleHandler" class)?

Note: I'm using emr-ami v. 4.6.0

Jonathan Kelly Jonathan Kelly · Accepted Answer · 2016-07-06T19:08:26

The yarn.nodemanager.aux-services property only really needs to be set on nodes that run the YARN NodeManager, which on EMR are only the CORE/TASK instances and not the MASTER instance (unless it is a single-node cluster).

On EMR the yarn.nodemanager.aux-services and yarn.nodemanager.aux-services.spark_shuffle.class properties are indeed set properly on the CORE/TASK instances though not on the MASTER instance, so the AWS EMR documentation is correct in that all you need to do to enable dynamicAllocation is to set spark.dynamicAllocation.enabled=true (either when creating your cluster or for an individual application via spark-submit options).

In fact, dynamicAllocation has been enabled by default as of emr-4.4.0, so you actually don't need to do/configure anything at all in order to use dynamicAllocation on emr-4.4.0+. (The one exception is if you have enabled the maximizeResourceAllocation feature. You may still use dynamicAllocation along with maximizeResourceAllocation, but you will need to enable dynamicAllocation explicitly in the configuration when creating your cluster in order to prevent spark.executor.instances from being set by maximizeResourceAllocation, since setting the number of executor instances effectively disables dynamicAllocation.)

By the way, you are correct in saying that the MASTER instance does have yarn.nodemanager.aux-services set only to mapreduce_shuffle, but this only is an artifact of how the configuration for this value is set on EMR. Though it can be confusing to see this value seemingly set incorrectly on the MASTER instance (i.e., missing the spark_shuffle value), it actually has no effect because the MASTER instance does not run a NodeManager.

Also, your assumption about Spark falling back to mapreduce_shuffle since spark_shuffle is not present is not correct. The MapReduce Shuffle Service is used only by MapReduce applications and cannot be used by Spark applications.

EMR configuration for spark.dynamicAllocation doesn't match Spark official documentation

1 Answers