Utilize multiple executors and workers in Spark job

Question

I am running spark on a standalone mode with below spark-env configuration -

export SPARK_WORKER_INSTANCES=4
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=4g

With this I can see 4 workers on my spark UI 8080.

Now One thing is the number of executors on my master URL (4040) is just one, how can I increases this to say 2 per worker node.

Also when I am running a small code from spark its just making use of one executer, do I need to make any config change to ensure multiple executors on multiple workers are used.

Any help is appreciated.

Answer 1

Set spark.master parameter as local[k], where k is the number of threads you want to utilize. You'd better to write these parameters inside spark-submit command instead of using export.

Answer 2

Parallel processing is based on number of partions of RDD. If your Rdd has multiple partions then it will processed parallelly.

Do some Modification (repartion) in your code, it should work.