Kinesis to Lambda - Parsing Data

Question

I'm following this tutorial to push data from my API Gateway to a Kinesis stream :

http://docs.aws.amazon.com/apigateway/latest/developerguide/integrating-api-with-aws-services-kinesis.html#api-gateway-get-and-add-records-to-stream

I have my Body Mapping Template setup as.....

{
    "StreamName": "my-stream-name",
    "Data": "$util.base64Encode($input.path('$.Data'))",
    "PartitionKey": "$input.path('$.PartitionKey')"
}

...and have put the following in the Request Body of an API test...

{
  "Data": {
    "Foo": "A",
    "Bar": "B"
  },
  "PartitionKey": "some key"
}

I've then created a Lambda Function which has a trigger set up against the same Kinesis Stream. However, I'm struggling to decode/deserialise the records coming in from Kinesis.

exports.handler = (event, context, callback) => {    
    event.Records.forEach(function(record) {        
        let payload = JSON.parse(Buffer(record.kinesis.data, 'base64').toString('ascii'))      
    });
};

It seems that the data is being serialised to Kinesis in a non-JSON format. The value for record.kinesis.data in the forEach loop is

e0Zvbz1BLCBCYXI9Qn0=

...which when push through Buffer(record.kinesis.data, 'base64').toString('ascii')

returns as

{Foo=A, Bar=B}

not

{"Foo":"A", "Bar":"B"}

Main aim is obviously to get payload to in a state where I can say console.log(payload.Foo)

Any hints as to what I should be doing/looking for would be appreciated.

ETFairfax ETFairfax · Accepted Answer · 2017-04-21T11:05:05

For anyone else out there

I had my Body Mapping Template setup as.....

{
    "StreamName": "my-stream-name",
    "Data": "$util.base64Encode($input.path('$.Data'))",
    "PartitionKey": "$input.path('$.PartitionKey')"
}

Now changed to handle the json....

{
    "StreamName": "my-stream-name",
    "Data": "$util.base64Encode($input.json('$.Data'))",
    "PartitionKey": "$input.path('$.PartitionKey')"
}

where...

($input.json('$.Data'))

is the change :)

Kinesis to Lambda - Parsing Data

1 Answers