Spring DataFlow Yarn - Container is running beyond physical memory

Question

I'm running Spring Cloud Tasks on Yarn simple tasks work fine but running bigger tasks which require more resources I got "Container is running beyond physical memory" error:

onContainerCompleted:ContainerStatus: [ContainerId: 
container_1485796744143_0030_01_000002, State: COMPLETE, Diagnostics: Container [pid=27456,containerID=container_1485796744143_0030_01_000002] is running beyond physical memory limits. Current usage: 652.5 MB of 256 MB physical memory used; 5.6 GB of 1.3 GB virtual memory used. Killing container.
Dump of the process-tree for container_1485796744143_0030_01_000002 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 27461 27456 27456 27456 (java) 1215 126 5858455552 166335 /usr/lib/jvm/java-1.8.0/bin/java -Dserver.port=0 -Dspring.jmx.enabled=false -Dspring.config.location=servers.yml -jar cities-job-0.0.1.jar --spring.datasource.driverClassName=org.h2.Driver --spring.datasource.username=sa --spring.cloud.task.name=city2 --spring.datasource.url=jdbc:h2:tcp://localhost:19092/mem:dataflow 
        |- 27456 27454 27456 27456 (bash) 0 0 115806208 705 /bin/bash -c /usr/lib/jvm/java-1.8.0/bin/java  -Dserver.port=0 -Dspring.jmx.enabled=false -Dspring.config.location=servers.yml -jar cities-job-0.0.1.jar --spring.datasource.driverClassName='org.h2.Driver' --spring.datasource.username='sa' --spring.cloud.task.name='city2' --spring.datasource.url='jdbc:h2:tcp://localhost:19092/mem:dataflow' 1>/var/log/hadoop-yarn/containers/application_1485796744143_0030/container_1485796744143_0030_01_000002/Container.stdout 2>/var/log/hadoop-yarn/containers/application_1485796744143_0030/container_1485796744143_0030_01_000002/Container.stderr

I tried tuning options in DataFlow's server.yml settings:

spring:
 deployer:
   yarn:
     app:
       baseDir: /dataflow
       taskappmaster:
         memory: 512m
         virtualCores: 1
         javaOpts: "-Xms512m -Xmx512m"
       taskcontainer:
         priority: 1
         memory: 512m
         virtualCores: 1
         javaOpts: "-Xms256m -Xmx512m"

I found out that taskappmaster memory changes are visible (AM container in YARN is set to this value), but taskcontainer memory options isnt changing - every container for Cloud Task which is created has only 256 mb which is default option for YarnDeployer.

For this server.yml expected result is allocation of 2 containers with 512 both for Application Master and Application Container. But YARN allocates 2 containers 512 for application master and 256 mb for application.

I dont think this problem is connected with YARN wrong options because Spark Applications work correctly seizing GBs of memory.

Some of my YARN settings:

mapreduce.reduce.java.opts -Xmx2304m
mapreduce.reduce.memory.mb 2880
mapreduce.map.java.opts -Xmx3277m
mapreduce.map.memory.mb 4096
yarn.nodemanager.vmem-pmem-ratio 5
yarn.nodemanager.vmem-check-enabled false
yarn.scheduler.minimum-allocation-mb 32
yarn.nodemanager.resource.memory-mb 11520

My Hadoop runtime is EMR 4.4.0 also I had to change default java to 1.8.

Did you remember to destroy pushed apps from hdfs under /dataflow when you modified that yml file? That file in hdfs will not get overridden. — Janne Valkealahti
Thank you very much, removing servers.yml from hdfs resolved my issue! — Ragnar
Hi, @Ragnar: Since this is resolved after the directory cleanup and successful bootstrap of the server, could you please consider adding an answer to this thread and mark it resolved? — Sabby Anandan

Ragnar Ragnar · Accepted Answer · 2017-02-04T16:50:02

Cleaning up /dataflow directory in HDFS resolves problem, after deleting this directory Spring DataFlow upload all needed files. The other way is to remove file by yourself and upload new one.

Spring DataFlow Yarn - Container is running beyond physical memory

1 Answers