How should I configure app engine task queue to maximize processing speed within rate limit

Question

I'm using Google app engine standard (Php) with task queues to process email sending for 15,000 recipients for a single message via Amazon Simple email service (SES). Imposed by SES is a rate limit of 14 messages per second which is the maximum rate I could get at this time.

I'm trying to configure my app.yaml and queue.yaml files to maximize the speed of getting a single email broadcast out to all its recipients while staying within the rate limit.

I have a handler setup to calculate the recipients of each message then it pushes a single task per recipient to another handler that personalizes the message and sends it using the SES api. I need some guidance on what to use as scaling parameters, maximum concurrent requests, bucket size, and rate. From what I understand it's not as simple as setting the task queues processing rate at 14/s.

Answer 1

I'd simply tweak the first handler to:

• only enqueue 14 personalize & send tasks, with no ETA
• enqueue another instance of itself, with a countdown of 1 second or, for even tighter "packing", with an ETA exactly 1 second after the moment it started its execution

At its next invocation, minimum 1 second later, it'll enqueue another 14 tasks and another instance of itself with ETA, etc., until there are no more addresses to send to. Maybe using queries with cursors to obtain those 14 addresses at each invocation, for example.

I'd use the same queue for both kinds of tasks, to ensure the main handler tasks are never processed before all of the 14 personalize & send tasks enqueue by its predecessor instance started processing.

With such approach you don't need a very precise queue configuration - meeting the SES max rate requirement should be guaranteed by the scheme itself, it just needs a minimum rate configured to allow the scheme to work.

It's worth noting that you should take special care for task failure/retry. Since you're already sending at max rate, any such retry would cause exceeding the max SES rate. So you want to disable such retries, you need to keep track of the addresses for which sending failed and handle them after the main sending sequence finishes.