How to load extra spark properties using --properties-file option in spark yarn cluster mode?

3
votes

I'm running a spark job using spark-submit in yarn cluster mode. To submit input and output file paths at run time, I'm trying to load a property file which contains the input and output paths.

Property file:input.properties


    spark.myapp.input /input/path
    spark.myapp.output /output/path

I'm running my application using below command.

  `spark-submit --class Property --master yarn-cluster prop.jar --properties-file input.properties`

Scala Code:


    import org.apache.spark.SparkConf
    import org.apache.spark.SparkContext
    import java.io.FileInputStream
    import collection.JavaConversions._
    import java.util.Properties;
    object Property {
      def main(args: Array[String]) {
        val conf = new SparkConf().setAppName("myApp");
        val sparkContext = new SparkContext(conf);
        val input=sparkContext.getConf.get("spark.myapp.input")
        println(input)
        sparkContext.stop;
      }
    }

I'm able to access these properties when I run my program in local and yarn-client mode. But in spark-submit mode, I'm getting below exception.

    ERROR yarn.ApplicationMaster: User class threw exception: java.util.NoSuchElementException: spark.myapp.input
2
scalaapache-sparkyarn

2 Answers

2
votes

use --files input.properties in case you are using yarn. I had same issue and it solved mine.

0
votes

I use the --files and --driver-class-path arguments to read the properties in on the driver that is executing on the remove Yarn cluster.

spark-submit \
    --class com.acme.Main \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 2g \
    --executor-memory 1g \
    --driver-class-path "./conf" \
    --files "./conf/app.properties,./conf/log4j.properties" \
    ./lib/my-app-uber.jar \
    "$@"

Note that multiple properties files can be specified using a comma delimited string (don't forget the quotes)

Your driver code can then load these properties files as classpath resources the same as if it was running locally.