Spark Memory allocation in YARN for apache zeppelin

1
votes

I run Spark through Zeppelin in a YARN cluster. When I see my YARN web ui I see that Zeppelin uses 1 container, 1 core and 1g of memory. So go to my interpreter settings and I change them to:

spark.cores.max 8 
spark.executor.memory 8g

and I restart the interpreter but my memory allocation stays the same: enter image description here

I event tried to add to interpreter property args to:

args --driver-memory 8G --executor-memory 8G --executor-cores 8

but no change either.

Appreciate any suggestion.

2
apache-sparkyarnapache-zeppelin
Have you specified "yarn.scheduler.capacity.resource-calculator":"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator" in capacity-scheduler.xml? - Glennie Helles Sindholt
Turn out that Zeppelin does not request the memory from YARN until it is necessary. I will later answer this question or delete this post. - Michail N

2 Answers

1
votes

Two Options

  1. edit $ZEPPELIN_HOME/conf/zeppelin-env.sh, add export SPARK_SUBMIT_OPTIONS="--num-executors 10 --driver-memory 8g --executor-memory 10g --executor-cores 4 ".

    • if you don't have zeppelin-env.sh, please copy and rename zeppelin-env.sh.template to zeppelin-env.sh.

  2. edit $SPARK_CONF_DIR/spark-defaults.conf and modify what you want to add.

After that, restart your server.

1
votes

This is a bug of zeppelin 0.7.x, 0.8.0 has fixed this issue.