YARN "Memory Used" in RM UI twice as spark-shell requested

Question

spark-shell started using:

spark-shell --master yarn --executor-memory 4G --num-executors 100

I'm expecting yarn to assign about 400GB memory to spark-shell, but when I go to RM UI, it's showing about 804GB increment of "Memory Used".

I'm running HDP 2.5 with yarn.scheduler.minimum-allocation-mb set to 4096 in yarn-site.xml.

Confused about how this is happening.

---

It turns out to be a problem about spark memory overhead and yarn memory allocation mechinism, check:

http://www.wdong.org/spark-on-yarn-where-have-all-the-memory-gone.html

Rule 1. Yarn always rounds up memory requirement to multiples of yarn.scheduler.minimum-allocation-mb, which by default is 1024 or 1GB. That’s why the driver’s requirement of 4G+384M showed up as 5G in Yarn. The parameter yarn.scheduler.minimum-allocation-mb is really “minimum-allocation-unit-mb”. This can be easily verified by setting the parameter to a prime number, such as 97, and see Yarn allocate by multiples of the number.

Rule 2. Spark adds an overhead to SPARK_EXECUTOR_MEMORY/SPARK_DRIVER_MEMORY before asking Yarn for the amount.

That is, in my configuration, even spark-shell asks for only 384MB overhead memory, which is default for spark.yarn.executor.memoryOverhead, yarn will allocate another 4GB for it.

Answer 1

My understanding is that --executor-memory gives you the upper bound for the memory an executor could use ever and becomes an executor's -Xmx (of the JVM).

In a particular point in time, it may use less (which is good).

I'd go to YARN nodes and do jps -lm and see the properties the executor JVMs use.

Answer 2

I came across the same issue in our environment where yarn is set to allocate 12 GB per core. Spark users where submitting jobs with --executor-memory 12g trying to utilize the allocated memory. However, this was causing yarn to allocate another 12 GB for each core. For example a 200 core hive query would use ~ 2.5 TB of memory, but a 200 core spark job submitted with driver memory and executor memory set to 12 GB would use almost 5 TB of memory.

When an application requests more memory than is available yarn allocates another "unit" of memory. So our spark jobs had to be requesting more than 12 GB total to trigger this additional allocation. As the post you link mentions, the overhead memory for spark is set to the max of 384 MB or .07 * executor memory. Therefore, in our specific case each executor was actually requesting 12 + (12 * 0.07) = 12.84 GB. As this is greater than the unit allocated by yarn, each core then received a second 12 GB unit of memory. The workaround for this was to decrease the driver and executor memory to be low enough such that the memory and overhead total less than 12 GB. In our case setting --driver-memory and --executor-memory to 11150m did the trick (you could also use 11g for simplicity)