Pass file as command line argument to Spark

Question

I am coding a Spark job in Scala and need to send some argument through command-line in JSON file format like the application name, master and some more variables.

./bin/spark-submit --name "My app" --master local[4] --conf spark.eventLog.enabled=false --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails -XX:+PrintGCTimeStamps" myApp.jar

I need to send app name, master and all arguments in one JSON file like:

$SPARK_HOME/bin/spark-submit --properties-file  property.conf

Is that possible? How? Can anyone please explain with a simple example?

You can add those configuratons to conf/spark-defaults.conf in the form of k-v pairs separated by a space, and spark-submit will read it from there — philantrovert
I am using github.com/scopt/scopt which is really nice for parsing command line argument. — koiralo
thanks philantrovert , default is good , but i need to override default configration using command line right now i am using json string as argument but its little bit difficult so need to parse as file or object ,may be jar is good option but every time i change in configuration file than i need to change in jar to at this time i need to build jar every time ??? — Hardik

stefanobaghino stefanobaghino · Accepted Answer · 2017-07-06T08:30:10

You can use the --jars option as follows:

$SPARK_HOME/bin/spark-submit --jars property.conf --class your.Class your.jar

The help page of spark-submit will yell you more:

$SPARK_HOME/bin/spark-submit --help

  --jars JARS Comma-separated list of local jars to include on the driver
              and executor classpaths.

Despite the name, you can also use it to move around configuration files that you want to be in your driver and executors' classpath.

Pass file as command line argument to Spark

1 Answers