Spark worker memory

9
votes

I have setup a spark (1.6)standalone cluster. have 1 master and added 3 machines under conf/slaves file as workers. Even though I have allocated 4GB memory to each of my workers in spark, why does it use only 1024MB when the application is running? I would like for it use all 4 GB allocated to it. Help me figure out where and what I am doing wrong.

Below is the screenshot of the spark master page (when the application is running using spark-submit) where under the Memory column it shows 1024.0 MB used in brackets next to 4.0 GB.

I also tried setting --executor-memory 4G option with spark-submit and it does not work (as suggested in How to change memory per node for apache spark worker).

These are the options I have set in spark-env.sh file

export SPARK_WORKER_CORES=3

export SPARK_WORKER_MEMORY=4g

export SPARK_WORKER_INSTANCES=2

enter image description here

3
apache-spark

3 Answers

5
votes

One other workaround is try setting the following parameters inside the conf/spark-defaults.conf file:

spark.driver.cores              4
spark.driver.memory             2g
spark.executor.memory           4g

Once you set the above (only the last line in your case) shut down all the workers at once and restart them. It is better to initialize the executors memory this way since your problem seems to be that no executor can allocate all the available memory of his worker.

1
votes

The parameter you are looking for is executor-memory try providing that to your spark application when you start it.

--executor-memory 4g

When you set worker-memory to 4g then the biggest executor you run on that worker is of 4g. PS: you can have different configurations (each having different worker memory).

0
votes

create a file called spark-env.sh in spark/conf directory add this line SPARK_EXECUTOR_MEMORY=4g