0
votes

From the amazon develop guide:

Shards are ephemeral: They are created and deleted automatically, as needed. Any shard can also split into multiple new shards; this also occurs automatically. (Note that it is also possible for a parent shard to have just one child shard.) A shard might split in response to high levels of write activity on its parent table, so that applications can process records from multiple shards in parallel.

In my application, I don't really care which shard data comes from, I just want to process all changes. When you set up your stream processors, though, you have to subscribe to a shard or set of shards. How can you be sure you're getting all the changes if shards can be removed/split up/added at any time?

Should a client poll the list of shards for changes?

1

1 Answers

1
votes

Yes. If you are using the low-level APIs of DynamoDB streams, you need to take care of these scenarios on your own.

A better idea is to use the DynamoDB Streams Kinesis Adapter, which allows you to use the Amazon Kinesis Client Library. The latter takes care of all the above-mentioned complexities for you, thereby making your life simpler.

Relevant documentation can be found here.