Where to put application property file for spark application running on AWS EMR

1
votes

I am submitting one spark application job jar to EMR, and it is using some property file. So I can put it into S3 and while creating the EMR I can download it and copy it at some location in EMR box if this is the best way how I can do this while creating the EMR cluster itself at bootstrapping time.

1
amazon-web-servicesapache-sparkamazon-emr
Possible duplicate of this : stackoverflow.com/questions/47535424/… – user238607
It’s not duplicate of that I want to know how to do this while creating the Emr cluster itself – Girish
I personally store my applications on s3 and load them on bootstrap. But I'm not sure if this is the answer that you are looking for... – eliasah
If you're running your job as an EMR step with spark-submit, you can reference the file directly in S3: spark-submit .... --files s3://mybucket/my/config/file – Dan W

1 Answers

0
votes

Check following snapshot

enter image description here

In edit software setting you can add your own configuration or JSON file ( which stored on S3 location ) and using this setting you can passed configure parameter to EMR cluster on creating time. For more details please check following links

Amazon EMR Cluster Configurations

Configuring Applications

AWS ClI

hope this will help you.