Amazon EMR: Passing java system property to custom jar

1
votes

Does Amazon EMR allow passing a system property to a custom jar, e.g. hadoop jar -Dkey=value myjob.jar? (key, value above used during initialization of the application itself, rather then belong to hadoop's Configuration object.)

The related thread How to specify mapred configurations & java options with custom jar in CLI using Amazon's EMR? discusses ways to pass system properties to hadoop daemons only via Bootstrap Actions, which, apparently, won't allow do the same for the java entry point class.

1
javahadoopamazon-emrsystem-properties
If you even somehow able to do that but the <key,value> won't be accessible from all the mappers and reducers unless you set them in the context. – SSaikia_JtheRocker
That's precisely the goal. – Ivan Balashov
If that's the case, please check my answer - might help you. Thanks! – SSaikia_JtheRocker

1 Answers

1
votes

If you don't want to pass the to the mappers or reducers, you can do the following in the terminal or from a script -

export HADOOP_OPTS="-Dkey=value"
hadoop jar ...

You can also put those in $HADOOP_HOME/conf/hadoop-env.sh, if you want it for every job, without explicitly defining them every time you run a job.

Hope this makes sense.