How to add an EMR Spark Step?

1
votes

According to the docs:

For Step type, choose Spark application.

But in Amazon EMR -> Clusters -> mycluster -> Steps -> Add step -> Step type, the only options are:

enter image description here

3
amazon-web-servicesapache-sparkamazon-emr
what version of emr and which region? – Lamanus
@Lamanus region=eu-central-1 – Alon
@Lamanus where can I see my EMR version? – Alon
The emr image version something like emr-5.29.0 or emr-6.0.0 – Lamanus

3 Answers

2
votes

There are two ways to add EMR spark steps:

- Using command-runner.jar (custom application) enter image description here

spark-submit --class org.apache.spark.examples.SparkPi /usr/lib/spark/examples/jars/spark-examples.jar 10

Using aws cli to do the same

aws emr add-steps --cluster-id j-xxxxxxxx --steps Name="add emr step to run spark",Jar="command-runner.jar",Args=[spark-submit,--class,org.apache.spark.examples.SparkPi,/usr/lib/spark/examples/jars/spark-examples.jar,10]

- Spark Application enter image description here

1
votes

I don't have a Spark Application option because I created a Core Hadoop cluster.

When I created the cluster, under Software configuration, I should have chosen Spark, then I would have had the Spark application option under Step type.

1
votes

You can use command-runner.jar for your use case. For the step type let it be Custom Jar from the options that you have. Check out this image for detail.

command-runner usage

You can read more about command-runner.jar command-runner-usage