Why does Spark not allocate new YARN containers when current memory level is not sufficient?

1
votes

I have Cloudera Cluster With the YARN capacity of 600 Vcores and 3600 GiB of Memory. But Admin team has configured the maximum memory for yarn container to be 6 GB. My User has as permission to allocate as many containers as he can .

When i Try to run a spark Job on data set of size 50 Gb Job fails with error of Executor Memory Overhead.

When one container Memory isnt sufficient why cant spark try for new container?

1
apache-sparkhadoop-yarn

1 Answers

0
votes

When one container Memory isnt sufficient why cant spark try for new container?

...because Spark does not do this by default (and you have not configured it otherwise).

The number of executors and more importantly the total number of CPU cores and RAM memory is controlled by you at spark-submit time. That's where --driver-memory, --executor-memory, --driver-cores, --total-executor-cores, --executor-cores, --num-executors and others.

$ ./bin/spark-submit --help
...
  --driver-memory MEM         Memory for driver (e.g. 1000M, 2G) (Default: 1024M).
  --driver-java-options       Extra Java options to pass to the driver.
  --driver-library-path       Extra library path entries to pass to the driver.
  --driver-class-path         Extra class path entries to pass to the driver. Note that
                              jars added with --jars are automatically included in the
                              classpath.

  --executor-memory MEM       Memory per executor (e.g. 1000M, 2G) (Default: 1G).
...
 Spark standalone with cluster deploy mode only:
  --driver-cores NUM          Cores for driver (Default: 1).
...
 Spark standalone and Mesos only:
  --total-executor-cores NUM  Total cores for all executors.

 Spark standalone and YARN only:
  --executor-cores NUM        Number of cores per executor. (Default: 1 in YARN mode,
                              or all available cores on the worker in standalone mode)

 YARN-only:
  --driver-cores NUM          Number of cores used by the driver, only in cluster mode
                              (Default: 1).
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --num-executors NUM         Number of executors to launch (Default: 2).
                              If dynamic allocation is enabled, the initial number of
                              executors will be at least NUM.
...

Some are deploy mode-specific while others depend on the cluster manager in use (which would be YARN in your case).

Wrapping up...it's you to decide how many resources to assign to a Spark application using spark-submit's options.

Read up on Submitting Applications in the official documentation of Spark.