AWS EMR cancel step can not terminate the corresponding spark application

0
votes

I am designing the CI/CD for the long running Spark Jobs on AWS EMR. The spark job is added as one step in the EMR cluster. I am considering to use cancel step and add step for CI/CD. Therefore, I integrated a linux shell script in CI/CD pipeline, which is to cancel and add step. The step can be cancelled. However, the corresponding spark application is still running. Is it normal? What is the correct way to stop the running spark application for CI/CD? Thanks!

1
apache-sparkamazon-emr
you can login in to master node and do yarn application -kill application_id or you can use rest API curl -v -X PUT -d '{"state": "KILLED"}''localhost:8088/ws/v1/cluster/apps/${application_id}' ... please check about yarn rest API to change application state - kavetiraviteja
Thanks kavetiraviteja. It is a very good comment. I used this method to implement a lambda function for CICD purpose. it works well. The whole solution is as below. Step 1. Get the master dns name of the emr cluster by aws cli. Step 2. To check if the targeted spark job is running by using rest API. Step 3. To kill the spark job if it is running by using Rest API. Step 4. To use aws cli to add step to run spark job fetched from s3. Not sure if is the standard way for CICD for AWS EMR, but it works quite well. - yyuankm

1 Answers

1
votes

When the step is running, you cannot cancel through the AWS API, but you can do it by using EMR's yarn command such as

yarn application -kill application_id

where the id can obtain from the below command.

yarn application -list