AWS SQS and Lambda Function : How to fail a lambda function and force message back on to queue

Question

We have a Lambda event source that polls SQS for messages and passes them on to a Lambda function.

The SQS has a Visibility Timeout that dictates how long after a consumer (lambda) picks up a message it will re-appear in the queue if not successfully completed.

Is there a way from the Lambda function to force this to happen as soon as we detect an error?

For example, say Lambda has a timeout of 10 mins. The SQS Visibility Timeout will need to be longer than that.

But if the Lambda function detects a problem early on, and throws an error - this will fail the message, but we have to wait for the SQS Visibility Timeout before the message is available for other consumers to try again.

Is there a way for the Lambda function to tell SQS that the message has failed and it should go back on the queue immediately ?

Answer 1

If you're using the built-in AWS Lambda SQS integration, then simply throwing an exception (returning a non-success status from the Lambda function) will result in all the SQS messages in the event of that invocation being placed back in the queue immediately.

From the documentation:

When Lambda reads a batch, the messages stay in the queue but become hidden for the length of the queue's visibility timeout. If your function successfully processes the batch, Lambda deletes the messages from the queue. If your function is throttled, returns an error, or doesn't respond, the message becomes visible again. All messages in a failed batch return to the queue, so your function code must be able to process the same message multiple times without side effects.

Answer 2

Call the ChangeMessageVisibility API and set the visibility timeout of each failed message to 0 seconds. Then throw an exception.

If you received a batch of SQS messages and were able to successfully process some of them, then explicitly delete those (successful) messages from the underlying SQS queue, then set the visibility timeout of each remaining failed message to 0 seconds, then throw an exception.

Setting a message's visibility timeout to 0 terminates the visibility timeout for that message and makes the message immediately visible to other consumers.