Spark streaming on yarn - Container running beyond physical memory limits

Question

I'm running a spark streaming application on Yarn, It works well for several days and after that I encountered a problem, the error message from yarn list below:

Application application_1449727361299_0049 failed 2 times due to AM Container for appattempt_1449727361299_0049_000002 exited with exitCode: -104
For more detailed output, check application tracking page:https://sccsparkdev03:26001/cluster/app/application_1449727361299_0049Then, click on links to logs of each attempt.
Diagnostics: Container [pid=25317,containerID=container_1449727361299_0049_02_000001] is running beyond physical memory limits. Current usage: 3.5 GB of 3.5 GB physical memory used; 5.3 GB of 8.8 GB virtual memory used. Killing container.

And here is my memory configuration:

spark.driver.memory = 3g
spark.executor.memory = 3g
mapred.child.java.opts  -Xms1024M -Xmx3584M
mapreduce.map.java.opts -Xmx2048M
mapreduce.map.memory.mb  4096
mapreduce.reduce.java.opts  -Xmx3276M
mapreduce.reduce.memory.mb  4096

This OOM error is strange because I didn't maintain any data in memory since it's a streaming program, does anyone encountered the same question like it? Or who know what cause it?

Answer 1

Check the mem on the box/vm instance you're running it on. My guess is the host machine is red lining it.

...due to, it appears, over-allocating memory.

Where do you think the streaming gets executed? Regardless of whether you store anything there? Yup. memory. Not cats or dancing Viking either (add "e").

Guess what? You're allocating 7 GB of memory that is heavily weighted towards physical over virtual mem.

1. Check your logging, as that would have similar build up time.

2. What's spark.yarn.am.memory value?

3. Get your VM and container memory allocation in balance :)

Another thought is to adjust memoryOverhead so as physical & virtual can be more proportional