AWS EMR 4.0 - How can I add a custom JAR step to run shell commands

2
votes

I am trying to run shell commands using steps on EMR 4.0.0 and used this link for reference - http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html

But I want to know from where to put 'command-runner.jar' in 'JAR location' field http://i.stack.imgur.com/CRicz.png

I kept 'command-runner.jar' in AWS s3 and tried to load it from that location and in 'Arguments' gave s3 location of my 'example.sh' file and after adding step it failed giving this exception

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:139)
    at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:92)
    ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 8 more
1
hadoopamazon-web-servicesamazon-s3amazon-emr
so you point from the Jar Location ? did you try the CLI from the document you refer in your question ? – Frederic Henri
No I didn't. Used AWS console. does it matter? – Sohan Malegaonkar
no it should not matter, but if console does not work, its good to see if CLI works so if its general error or an error with the use of console – Frederic Henri
@Frédéric It didn't work with CLI as well!!!! Are you sure I am supposed to locate command-runner.jar from s3? – Sohan Malegaonkar

1 Answers

1
votes

To run a shell script stored in S3, use script-runner.jar, which is located at s3://elasticmapreduce/libs/script-runner/script-runner.jar.

command-runner.jar will work for local files only.