How to get the status of a Step in AWS EMR cluster?

Question

I am trying to create an EMR Cluster and executing a spark Job.

I need to initiate a thread so that when my job completes it let me knows that the step has been completed, since my EMR cluster will be up and running.

{
    AmazonElasticMapReduceClient emr = configureEMRClient();

    StepFactory stepFactory = new StepFactory();

    StepConfig enableDebugging = new StepConfig()
        .withName("Enable Debugging")
        .withActionOnFailure("TERMINATE_JOB_FLOW")
        .withHadoopJarStep(stepFactory.newEnableDebuggingStep());

    HadoopJarStepConfig runExampleConfig = new HadoopJarStepConfig()
        .withJar("command-runner.jar")
        .withArgs("spark-submit")
        .withArgs("--deploy-mode","cluster")
        .withArgs("--master","yarn")
        .withArgs("--class", extractorMainClass )
        .withArgs("--num-executors","3")
        .withArgs("--driver-memory","8g")
        .withArgs( resourceExtractorJar )
        .withArgs("st")
        .withArgs("ap");

    StepConfig customExampleStep = new StepConfig()
            .withName("Example Step")
            .withActionOnFailure("TERMINATE_JOB_FLOW")
            .withHadoopJarStep(runExampleConfig);

    Application spark = new Application().withName("Spark");

    RunJobFlowRequest request = new RunJobFlowRequest()
            .withName("Test")
            .withReleaseLabel("emr-5.5.0")
            .withSteps(enableDebugging, customExampleStep)
            .withApplications(spark)
            .withServiceRole("EMR_DefaultRole")
            .withJobFlowRole("EMR_EC2_DefaultRole")
            .withInstances(new JobFlowInstancesConfig()
                    .withEc2KeyName("keypair")
                    .withInstanceCount(2)
                    .withKeepJobFlowAliveWhenNoSteps(false)
                    .withMasterInstanceType("m3.xlarge")
                    .withSlaveInstanceType("m3.xlarge"));       

    RunJobFlowResult result = emr.runJobFlow(request);

}

To get the step status :

while(true) {
              DescribeClusterRequest desc = new DescribeClusterRequest()
                .withClusterId(jobFlowId);
              DescribeClusterResult clusterResult = emrClient.describeCluster(desc);
              com.amazonaws.services.elasticmapreduce.model.Cluster cluster = clusterResult.getCluster();

              DescribeStepRequest d = new DescribeStepRequest().withClusterId(jobFlowId);
              DescribeStepResult r = emrClient.describeStep(d);

              Step ss = r.getStep();

              String status = ss.getStatus().getState();

              System.out.printf("Status: %s\n", status);
              if(status.equals(ClusterState.TERMINATED.toString()) || status.equals(ClusterState.TERMINATED_WITH_ERRORS.toString())) {
               System.out.println("Terminated");
                  break;
              }
              try {
                TimeUnit.SECONDS.sleep(30);
              } catch (InterruptedException e) {
                e.printStackTrace();
              }
              // maybe other handle
            }

Getting the following error:

Exception in thread "main" com.amazonaws.services.elasticmapreduce.model.InvalidRequestException: Step id 'null' is not valid. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: InvalidRequestException; Request ID: 98e1b43e-a440-11e7-920b-158124595c35)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)

How can i get the status of the step inside an EMR cluster deployment?

abhi5800 abhi5800 · Accepted Answer · 2017-09-28T12:29:06

Answering the question, since i figured out:

    ListStepsResult steps = emrClient.listSteps(new ListStepsRequest().withClusterId(jobFlowId));
    StepSummary step = steps.getSteps().get(0);
    System.out.println(step.getStatus().getState());

How to get the status of a Step in AWS EMR cluster?

1 Answers