AWS Firehose transform Lambda function throws Lambda.MissingRecordId error for every record

Question

I'm using AWS firehose with an S3 backup and an S3 destination bucket, and it works great. The problem arises when I try to transform the data with a Lambda function.

I'm using the .NET AWS SDK, my Lambda function is written in C# and is using the included firehose transform example:

[assembly:LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.JsonSerializer))]

namespace LambdaFunctions
{
    public class Function
    {
        public KinesisFirehoseResponse FunctionHandler(KinesisFirehoseEvent evnt, ILambdaContext context)
        {
            context.Logger.LogLine($"InvocationId: {evnt.InvocationId}");
            context.Logger.LogLine($"DeliveryStreamArn: {evnt.DeliveryStreamArn}");
            context.Logger.LogLine($"Region: {evnt.Region}");

            KinesisFirehoseResponse response = new KinesisFirehoseResponse
            {
                Records = new List<KinesisFirehoseResponse.FirehoseRecord>()
            };

            foreach (KinesisFirehoseEvent.FirehoseRecord record in evnt.Records)
            {
                context.Logger.LogLine($"\tRecordId: {record.RecordId}");
                context.Logger.LogLine($"\t\tApproximateArrivalEpoch: {record.ApproximateArrivalEpoch}");
                context.Logger.LogLine($"\t\tApproximateArrivalTimestamp: {record.ApproximateArrivalTimestamp}");
                context.Logger.LogLine($"\t\tData: {record.DecodeData()}");

                // Transform data: For example ToUpper the data
                KinesisFirehoseResponse.FirehoseRecord transformedRecord = new KinesisFirehoseResponse.FirehoseRecord
                {
                    RecordId = record.RecordId,
                    Result = KinesisFirehoseResponse.TRANSFORMED_STATE_OK
                };

                transformedRecord.EncodeData(record.DecodeData().ToUpperInvariant());

                response.Records.Add(transformedRecord);
            }

            return response;
        }
    }
}

The data is successfully & correctly processed by the transform Lambda function (as indicated by tests & logs).

However, the data is not successfully returned by the Lambda function to the S3 destination bucket, all of the records are unsuccessfully processed.

This error is returned for each record:

{
    "attemptsMade": 1,
    "arrivalTimestamp": 1590656820209,
    "errorCode": "Lambda.MissingRecordId",
    "errorMessage": "One or more record Ids were not returned. Ensure that the Lambda function returns all received record Ids.",
    "attemptEndingTimestamp": 1590656883464,
    "rawData": "dGVzdDE=",
    "lambdaArn": "arn:aws:lambda:Region:AccountNumber:function:transform-function:$LATEST"
}

I'm at a loss as to why or where this error is occurring. I know the Lambda function is returning the correct response, recordId included.

I've recreated all the resources, applied and re-applied permissions, done just about everything I can think of.

This issue doesn't happen when using the Node.js or Python versions, it seems to be unique to the .NET implementation.

EDIT:

I forgot to add the serializer assembly attribute to the original code block which ended up being the source of the issue.

Jack Jack · Accepted Answer · 2020-05-28T13:41:31

The C# example provided by AWS is out of date, specifically this serialization package:

Amazon.Lambda.Serialization.SystemTextJson

To solve this issue, simply replace the package with this one:

Amazon.Lambda.Serialization.Json

and update the assembly attribute like so:

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]

namespace LambdaFunctions
{
    ...

This serialization package is mentioned in the AWS Documentation here, under "Serializing Lambda functions".

However, Amazon have not yet updated the SDK examples to reflect this change (or at least, this example specifically), causing the function to fail when deployed.

AWS Firehose transform Lambda function throws Lambda.MissingRecordId error for every record

1 Answers