Configuring Executor and Driver memory in Spark-on-Yarn

Question

I am confused with configuring executor and driver memory in Spark-1.5.2.

My environment settings are as below:

3 Node MAPR Cluster - Each Node: Memory 256G, 16 CPU 
Hadoop 2.7.0 
Spark 1.5.2 - Spark-on-Yarn

Input data information:

460 GB Parquet format table from Hive I'm using spark-sql for querying the hive context with spark-on-yarn,but it's lot slower than the Hive, and am not sure with the right memory configurations for Spark,

These are my config's,

    export SPARK_DAEMON_MEMORY=1g
    export SPARK_WORKER_MEMORY=88g

    spark.executor.memory              2g
    spark.logConf                      true
    spark.eventLog.dir                 maprfs:///apps/spark
    spark.eventLog.enabled             true
    spark.serializer                   org.apache.spark.serializer.KryoSerializer
    spark.driver.memory                5g
    spark.kryoserializer.buffer.max    1024m

How to avoid Spark java.lang.OutOfMemoryError: Java heap space exceptions and GC overhead limit exceeded exceptions!! ???

Really appreciate your assistance in this!

have you tried spark.executor.instances which will be the number of you spark workers in the cluster ? — dumitru

Luis Luis · Accepted Answer · 2017-07-12T15:26:30

At a first glance, you are running out the memory of your executors. I would suggest increasing their memory.

Note that SPARK_WORKER_MEMORY is only used in standalone mode. SPARK_EXECUTOR_MEMORY is used in YARN mode.

If you are not running anything else on the cluster you could try out the following config:

spark.executor.memory   16g
spark.executor.cores    1
spark.executor.instances 40
spark.driver.memory  5g (make it bigger if expected 
                         final result dataset is larger)

I do not recommend to set a large executor memory because that typically increments the GC time. Other thing I see, it is that those instances are memory optimized. Think twice if this fits your case.

Configuring Executor and Driver memory in Spark-on-Yarn

1 Answers