DynamoDB keep track of execution between 2 tables

Question

I've a Lambda which makes an API request to a service and get a list of matches for a given identifier. Once it gets a list of matches it updates a DynamoDB table in the following way:

requestId (partitionKey)	ID	matches	status
uuid1	ID1	[match1,match2, match3...]	FOUND_MATCHES

For each match in this table, a message is sent to SQS which is listened to by a Lambda. For each match, the Lambda will make a call to a different service and update a table which keeps track of the matches execution.

Match (partitionKey)	requestId (sortKey + GSI Partition Key)
match1	uuid1
match2	uuid1
...	...
match10	uuid1

Now given the requestId, I would like to know if there are entries in the 2nd table for all matches.

One option I'm thinking is look up the first table via the requestId and it will give the list of matches, then it will call the 2nd table multiple times via the primary key, sortKey combination and compile a result. The other option is instead of looking up via primary key/sort key for 2nd table, it will look via a GSI Parition key on the requestId column and get all the matches at once (in a paginated list).

I would like to expose this operation via an API call, and I'm wondering if I'll run into APIG 30 seconds timeout (I know I'll need to run some experiments with my dataset, but just wanted to see if there are other options I can consider before doing this). If number of matches exceed say 50,000. And each get call roughly takes 20ms, it will take about 50,000 * 20 = 1000 seconds which is way more than APIG limit. Maybe batch call might help a bit, but not sure there is much room there.

Essentially I would like to update the status in the first table from FOUND_MATCHES to ALL_MATCHES_PROCESSED.

1. Ideal option is automatically the state gets updated
2. A Get status API essentially triggers the calculation and updates the state (Maybe make this async to get past the 30seconds APIG limitation)

Answer 1

I'd probably do it like this, since I'm a fan of the single table design pattern:

PK	SK	type	attributes
REQ#<id>	META	REQUEST	matches: [a,b,c]; unprocessed_matches: 3; status: FOUND_MATCHES
REQ#<id>	M#a	MATCH	result
REQ#<id>	M#b	MATCH	result
REQ#<id>	M#c	MATCH	result

Whenever a new request comes in, you write a REQUEST to the table with the matches and an unprocessed_matches attribute that counts all matches.

Then you have a Lambda function "stream-processor" that listens to the stream of that table. When a new REQUEST item shows up (no old image present), it creates the tasks in your SQS queue.

The worker process on your SQS queue then calls the 3rd party API and records the result. The result is then written to the table through a transaction with two items:

1. An UpdateItem call, that decrements the unprocessed_matches value for the request id by one.
2. A PutItem call, that creates the MATCH item.

The transaction here is important, you want this to be an all or nothing operation.

Your stream-processor lambda then has a second job. Whenever there is update to a REQUEST item, it should check, if unprocessed_matches <= 0 and status = FOUND_MATCHES. When that's the case, it should update the REQUEST item and set the status to ALL_MATCHES_PROCESSED.

Your original Lambda that made the first request can periodically poll the status of the item in the table.

This design also let's you easily get everything about a request by a simple Query to the partition key with the request ID.

You should be aware that there is a 400KB item size limit in DynamoDB, depending on how long your matches list gets, this might become a problem.