How to config a specific node as a driver node in spark-submit with cluster mode and standalone mode

Question

I have to problem in spark-submit with cluster deploy mode and standalone mode:

• How to specify a node as a driver node in spark cluster

  in my case, the driver node was assigned dynamically by spark

• How to distribute the app automatic from local

  in my case, i must deploy the jar of app to every node,because i don't know which node will be the driver node .

PS : My submit command is : spark-submit --master spark://master_ip:6066 --class appMainClass --deploy-mode cluster file:///tmp/spark_app/sparkrun

Answer 1

The --deploy-mode flag determines if the job will be submitted in cluster or client mode.

In cluster mode all the nodes will act as executors. One node will submit the JAR and then you can track the execution using web UI. That particular node will also act as an executor.

In client mode, the node where the spark-submit is invoked will act as the driver. Note that this node will not execute the DAG as this it is designated as a driver for your cluster. All the other nodes will be executors. Again, Web UI will help to see the execution of jobs and other useful information like RDD partitions, cached RDDs size etc.