Empty dictionary on AnnotationConsolidation lambda event for aws Sagemaker

Question

I am starting to use aws sagemaker on the development of my machine learning model and I'm trying to build a lambda function to process the responses of a sagemaker labeling job. I already created my own lambda function but when I try to read the event contents I can see that the event dict is completely empty, so I'm not getting any data to read.

I have already given enough permissions to the role of the lambda function. Including: - AmazonS3FullAccess. - AmazonSagemakerFullAccess. - AWSLambdaBasicExecutionRole

I've tried using this code for the Post-annotation Lambda (adapted for python 3.6):

https://docs.aws.amazon.com/sagemaker/latest/dg/sms-custom-templates-step2-demo1.html#sms-custom-templates-step2-demo1-post-annotation

As well as this one in this git repository:

https://github.com/aws-samples/aws-sagemaker-ground-truth-recipe/blob/master/aws_sagemaker_ground_truth_sample_lambda/annotation_consolidation_lambda.py

But none of them seemed to work.

For creating the labeling job I'm using boto3's functions for sagemaker: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sagemaker.html#SageMaker.Client.create_labeling_job

This is the code i have for creating the labeling job:

def create_labeling_job(client,bucket_name ,labeling_job_name, manifest_uri, output_path):

    print("Creating labeling job with name: %s"%(labeling_job_name))

    response = client.create_labeling_job(
        LabelingJobName=labeling_job_name,
        LabelAttributeName='annotations',
        InputConfig={
            'DataSource': {
                'S3DataSource': {
                    'ManifestS3Uri': manifest_uri
                }
            },
            'DataAttributes': {
                'ContentClassifiers': [
                    'FreeOfAdultContent',
                ]
            }
        },
        OutputConfig={
            'S3OutputPath': output_path
        },
        RoleArn='arn:aws:myrolearn',
        LabelCategoryConfigS3Uri='s3://'+bucket_name+'/config.json',
        StoppingConditions={
            'MaxPercentageOfInputDatasetLabeled': 100,
        },
        LabelingJobAlgorithmsConfig={
            'LabelingJobAlgorithmSpecificationArn': 'arn:image-classification'
        },
        HumanTaskConfig={
            'WorkteamArn': 'arn:my-private-workforce-arn',
            'UiConfig': {
                'UiTemplateS3Uri':'s3://'+bucket_name+'/templatefile'
            },
            'PreHumanTaskLambdaArn': 'arn:aws:lambda:us-east-1:432418664414:function:PRE-BoundingBox',
            'TaskTitle': 'Title',
            'TaskDescription': 'Description',
            'NumberOfHumanWorkersPerDataObject': 1,
            'TaskTimeLimitInSeconds': 600,
            'AnnotationConsolidationConfig': {
                'AnnotationConsolidationLambdaArn': 'arn:aws:my-custom-post-annotation-lambda'
            }
        }
    )

    return response

And this is the one i have for the lambda function:

    print("Received event: " + json.dumps(event, indent=2))
    print("event: %s"%(event))
    print("context: %s"%(context))
    print("event headers: %s"%(event["headers"]))

    parsed_url = urlparse(event['payload']['s3Uri']);
    print("parsed_url: ",parsed_url)

    labeling_job_arn = event["labelingJobArn"]
    label_attribute_name = event["labelAttributeName"]

    label_categories = None
    if "label_categories" in event:
        label_categories = event["labelCategories"]
        print(" Label Categories are : " + label_categories)

    payload = event["payload"]
    role_arn = event["roleArn"]

    output_config = None # Output s3 location. You can choose to write your annotation to this location
    if "outputConfig" in event:
        output_config = event["outputConfig"]

    # If you specified a KMS key in your labeling job, you can use the key to write
    # consolidated_output to s3 location specified in outputConfig.
    kms_key_id = None
    if "kmsKeyId" in event:
        kms_key_id = event["kmsKeyId"]

    # Create s3 client object
    s3_client = S3Client(role_arn, kms_key_id)

    # Perform consolidation
    return do_consolidation(labeling_job_arn, payload, label_attribute_name, s3_client)

I've tried debugging the event object with:

    print("Received event: " + json.dumps(event, indent=2))

But it just prints an empty dictionary: Received event: {}

I expect the output to be something like:

    #Content of an example event:
    {
        "version": "2018-10-16",
        "labelingJobArn": <labelingJobArn>,
        "labelCategories": [<string>],  # If you created labeling job using aws console, labelCategories will be null
        "labelAttributeName": <string>,
        "roleArn" : "string",
        "payload": {
            "s3Uri": <string>
        }
        "outputConfig":"s3://<consolidated_output configured for labeling job>"
    }

Lastly, when I try yo get the labeling job ARN with:

    labeling_job_arn = event["labelingJobArn"]

I just get a KeyError (which makes sense because the dictionary is empty).

José R. Sánchez José R. Sánchez · Accepted Answer · 2019-08-02T10:28:09

I found the problem, I needed to add the ARN of the role used by my Lamda function as a Trusted Entity on the Role used for the Sagemaker Labeling Job.

I just went to Roles > MySagemakerExecutionRole > Trust Relationships and added:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::xxxxxxxxx:role/My-Lambda-Role",
           ...
        ],
        "Service": [
          "lambda.amazonaws.com",
          "sagemaker.amazonaws.com",
           ...
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

This made it work for me.

Empty dictionary on AnnotationConsolidation lambda event for aws Sagemaker

2 Answers