AWS Kinesis, concurrent Lambda processing with a guaranteed ordering

Question

I have a Lambda with an Event source pointed to a Kinesis Stream Consumer (with an arbitrary number of shards)

I would like to ensure that items in the stream with the same 'partition key' are processed by Lambda in sequence and not simultaneously. ( This is being used as the object's identity, and I don't want multiple Lambdas performing logic on the same object simultaneously.)

For example, if the items in the stream have partition keys:

1,2,1,3,4,1,2,1

If we take the order of processing to be left to right, Lambda would process an item with each of the partition keys 1,2, 3 and 4 concurrently. Then, when it has finished an item with a specific partition key it can start processing another one with that key.

Is this achievable in some way, without the use of a distributed lock that would make inefficient use of Lambda?

Thanks

Answer 1

Seems like I was tackling the problem in the wrong way. Lambda guarantees that within a shard, the Lambda instance is invoked on one batch at a time. Therefore, there is no need for a distributed lock as at worst there will be multiple records belonging to the same entity in the same batch and processing them in order can be managed in-memory within the Lambda function itself.

Reference from the AWS FAQs http://aws.amazon.com/lambda/faqs/

Q: How does AWS Lambda process data from Amazon Kinesis streams and Amazon DynamoDB Streams?

The Amazon Kinesis and DynamoDB Streams records sent to your AWS Lambda function are strictly serialized, per shard. This means that if you put two records in the same shard, Lambda guarantees that your Lambda function will be successfully invoked with the first record before it is invoked with the second record. If the invocation for one record times out, is throttled, or encounters any other error, Lambda will retry until it succeeds (or the record reaches its 24-hour expiration) before moving on to the next record. The ordering of records across different shards is not guaranteed, and processing of each shard happens in parallel.

Answer 2

Items with the same 'partition key' will be processed by Lambda in sequence for stream event source mapping.

Moreover, you can specify 'concurrent batches per shard' when creating Lambda trigger:

1. If 'concurrent batches per shard' is 1 (default one), then the order will be preserved for the whole shard.
2. If 'concurrent batches per shard' is [2;10], then the order will be preserved only for records with the same partition key within the shard.

You can check about concurrent batches (ParallelizationFactor) in https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html