Load props file in EMR Spark Application

2
votes

I am trying to load custom properties in my spark application using :-

command-runner.jar,spark-submit,--deploy-mode,cluster,--properties-file,s3://spark-config-test/myprops.conf,--num-executors,5,--executor-cores,2,--class,com.amazon.Main,#{input.directoryPath}/SWALiveOrderModelSpark-1.0-super.jar

However, I am getting the following exception:-

Exception in thread "main" java.lang.IllegalArgumentException: Invalid properties file 's3://spark-config-test/myprops.conf''. at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241) at org.apache.spark.launcher.AbstractCommandBuilder.loadPropertiesFile(AbstractCommandBuilder.java:284) at org.apache.spark.launcher.AbstractCommandBuilder.getEffectiveConfig(AbstractCommandBuilder.java:264) at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:233) at org

Is this the correct way to load file from S3?

1
apache-sparkemramazon-data-pipeline

1 Answers

4
votes

You can't load a properties file directly from S3. Instead you will need to download the properties file to your master node somewhere, then submit the spark job referencing the local path on that node. You can do the download by using command-runner.jar to run the aws cli utility.