Java heap size error when running Spark from Python

1
votes

I'm trying to run a Python script with the pyspark library. I create a SparkConf() object using the following commands:

conf = SparkConf().setAppName('test').setMaster(<spark-URL>)

When I run the script, that line runs into an error:

Picked up _JAVA_OPTIONS: -Xmx128m

Picked up _JAVA_OPTIONS: -Xmx128m

Error occurred during initialization of VM Initial heap size set to a larger value than the maximum heap size.

I tried to fix the problem by setting the configuration property spark.driver.memory to various values, but nothing changed.

What is the problem and how can I fix it?

Thanks

1
javapythonapache-sparkpyspark
Try to provide good amount of memory for spark.driver.memory and spark.executor.memory - mrsrinivas

1 Answers

0
votes

This is because you're setting the maximum available heap size (128M) to be larger than the initial heap size error. Check the _JAVA_OPTIONS parameter that you're passing or setting in the configuration file. Also, note that the changes in the spark.driver.memory won't have any effect because the Worker actually lies within the driver JVM process that is started on starting spark-shell and the default memory used for that is 512M.

This creates a conflict as spark tries to initialize a heap size equal to 512M, but the maximum allowed limit set by you is only 128M.

You can set the minimum heap size through the --driver-java-options command line option or in your default properties file