Failed cron job handling with elastic beanstalk and SQS

Question

I have two elastic beanstalk environments.

One is the 'primary' web server environment and the other is a worker environment that handles cron jobs.

I have 12 cron jobs, setup via a cron.yaml file that all point at API endpoints on the primary web server.

Previously my cron jobs were all running on the web server environment but of course this created duplicate cron jobs when this scaled up.

My new implementation works nicely but where my cron jobs fail to run as expected the cron job repeats, generally within a minute or so.

I would rather avoid this behaviour and just attempt to run the cron job again at the next scheduled interval.

Is there a way to configure the worker environment/SQS so that failed jobs do not repeat?

Answer 1

Simply configure a CloudWatch event to take over your cron, and have it create an SQS message ( either directly or via a Lambda function ).

Your workers will now just have to handle SQS jobs and if needed, you will be able to scale the workers as well.

http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html

Answer 2

Yes, you can set the Max retries parameter in the Elastic Beanstalk environment and the Maximum Receives parameter in the SQS queue to 1. This will ensure that the message is executed once, and if it fails, it will get sent to the dead letter queue.

With this approach, your instance may turn yellow if there are any failed jobs, because the messages would end up in the dead letter queue, which you can simple observe and ignore, but it may be annoying if you are OCD about needing all environments to be green. You can set the Message Retention Period parameter for the dead letter queue to something short so that it will go away sooner though.

An alternative approach, if you're interested, is to return a status 200 OK in your code regardless of how the job ran. This will ensure that the SQS daemon deletes the message in the queue, so that it won't get picked up again.

Of course, the downside is that you would have to modify your code, but I can see how this would make sense if you don't care about the result.

Here's a link to AWS documentation that explains all of the parameters.