Rate-limiting a Worker for a Queue (e.g.: SQS)

Question

Every day, I will have a CRON task run which populates an SQS queue with a number of tasks which needs to be achieved. So (for example) at 9AM every morning, and empty queue will receive ~100 messages that will need to be processed.

I would like a new worker to be spun up every second until the queue is empty. If any task fails, it's put at the back of the queue to be re-run.

For example, if each task takes up to 1.5 seconds to complete:

after 1 second, 1 worker will have started message A
after 2 seconds, 1 worker may still be running message A and 1 worker will have started running message B
after 100 seconds, 1 worker may still be running message XX and 1 worker will pick up message B because it failed previous
after 101 seconds, no more workers are propagated until the next morning

Is there any way to have this type of infrastructure configured within AWS lambda?

This is an interesting use-case. Can you give us a sense for why the 1 second rate limiting would be necessary (even in broad terms)? It can be accomplished with an Executor that spawns 1 thread per second and processes exactly 1 SQS poll and then works a failed queue if not empty -- but I'm still curious as to where it would be desirable. Thanks! — Dave Maple
We're using it to communicate with a third-party API who rate limits our use of their service with a maximum of 1 request per second. — bashaus
I have the same use case: Intercom rate limits to 83 requests per 10s. The system's cron kicks off a batch job which emits onto a SQS queue and a worker eventually makes requests to Intercom — otupman

Andy Hayden Andy Hayden · Accepted Answer · 2017-10-09T00:17:57

One way, though I'm not convinced it's optimal:

A lambda that's triggered by an CloudWatch Event (say every second, or every 10 seconds, depending on your rate limit). Which polls SQS to receive (at most) N messages, it then "fans-out" to another Lambda function with each message.

Some pseudo code:

# Lambda 1 (schedule by CloudWatch Event / e.g. CRON)
def handle_cron(event, context):
    # in order to get more messages, we might have to receive several times (loop)
    for message in queue.receive_messages(MaxNumberOfMessages=10):
        # Note: the Event InvocationType so we don't want to wait for the response!
        lambda_client.invoke(FunctionName="foo", Payload=message.body, InvocationType='Event')

and

# Lambda 2 (triggered only by the invoke in Lambda 1)
def handle_message(event, context):
    # handle message
    pass

Rate-limiting a Worker for a Queue (e.g.: SQS)

3 Answers