How to pass Kinesis Firehose data to dynamodb table?

Question

A Kinesis Firehose stream receives messages.
There is an option to persist into S3, but my use case is to insert into dynamodb table.
Firehose has an option to enable Lambda function. Shall i write insert logic into dynamodb table using Lambda? Is this the right approach?
If so, then how to insert records into DynamoDB using Lambda written in Java.

Horatiu Jeflea Horatiu Jeflea · Accepted Answer · 2019-10-12T07:21:11

There isn't a standard way of inserting Firehose stream data into DynamoDB (such as S3 or Redshift). The recommended way is to do a Lambda and insert the records into DynamoDB with that.

Use dynamoDB.batchWriteItem or dynamoDB.putItem, more info in this article or this one.

public String handleRequest(KinesisFirehoseEvent event, Context context)
    List<KinesisFirehoseEvent.Record> records = event.getRecords();
    for(KinesisFirehoseEvent.Record rec : records)
    {
        String recordId = rec.getRecordId();
        String data = StandardCharsets.UTF_8.decode(rec.getData()).toString();

        Item item = transformStringToItem(data);
        // Write the item to the table 
        table.putItem(item);
    }
    return "success";
}

The firehoseStream structure is documented here, and a Java example example is here, and more info on firehose to lambda here.

How to pass Kinesis Firehose data to dynamodb table?

1 Answers