My application connects to a spark cluster (just starting a master and some slaves, not used yarn), now it throws StackOverwflowErr for ObjectInputStream, so i should increase -Xss of driver and executor. The documents said you can't change driver properties in your application code (using sparkConf).
changing spark-defaults.conf not worked (i guesss it only works with spark-submit) So how should i change driver related configs?
I just used: conf.set("spark.executor.extraJavaOptions","-Xss4m") in my java code and now spark executor run with xss=4m. I was biased with this part of spark documentation:
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.
