delete_message_batch doesn't really delete messages from SQS queue

Question

I am using a standard Amazon SQS queue. Using boto3 library in python3 to interact with SQS. Following is my code for receving messages and then deleting them:

from boto3.session import Session
boto3_session = Session(region_name=SQS_REGION_NAME, aws_access_key_id=SQS_ACCESS_ID,
                                aws_secret_access_key=SQS_ACCESS_KEY)
sqs = boto3_session.client('sqs')

response = sqs.receive_message(
    MessageAttributeNames=[
        'EventToReport',
    ],
    QueueUrl=queue_url,
    MaxNumberOfMessages=10,
    VisibilityTimeout=0,
    WaitTimeSeconds=0
)
messages = response['Messages']
receipt_handles = [{'Id': str(index), 'ReceiptHandle': msg['ReceiptHandle']} for index, msg in enumerate(messages)]
sqs.delete_message_batch(QueueUrl=queue_url, Entries=receipt_handles)

This returns a success response:

{'Successful': [{'Id': '0'}, {'Id': '1'}, {'Id': '2'}, {'Id': '3'}, {'Id': '4'}, {'Id': '5'}, {'Id': '6'}, {'Id': '7'}, {'Id': '8'}, {'Id': '9'}], 'ResponseMetadata': {'RequestId': 'bb28855b-6522-5a1e-a649-d7b3fdabfebe', 'RetryAttempts': 0, 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-length': '1008', 'connection': 'keep-alive', 'server': 'Server', 'date': 'Mon, 29 Jan 2018 03:34:33 GMT', 'content-type': 'text/xml', 'x-amzn-requestid': 'bb28855b-6522-5a1e-a649-d7b3fdabfebe'}}}

But when I look at my SQS management console, I can see that none of the messages are deleted! Visibility timeout for messages was 30 seconds.

I increased the visibility timeout of the queue to 30 minutes, and now only 1 message got deleted although the response was same!

I know it takes time for the delete to complete, so i waited 2-3 minutes every time, but the messages remain in the queue... Am I calling the API in a wrong way? As far as I could find, this is the correct way to call sqs api using boto3.

Answer 1

I finally found the problem with my code. Although I was describing visibility timeout in sqs console, my get request overrode that, so I just removed the following 2 lines from my receive_message code:

VisibilityTimeout=0,
WaitTimeSeconds=0

It is working fine now.

Answer 2

How SQS works with deletion requests

There is a description of why the settings VisibilityTimeout and WaitTimeSeconds affect the behavior of deletion methods.

Note

The ReceiptHandle is associated with a specific instance of receiving a message. If you receive a message more than once, the ReceiptHandle is different each time you receive a message. When you use the DeleteMessage action, you must provide the most recently received ReceiptHandle for the message (otherwise, the request succeeds, but the message might not be deleted).

For standard queues, it is possible to receive a message even after you delete it. This might happen on rare occasions if one of the servers which stores a copy of the message is unavailable when you send the request to delete the message. The copy remains on the server and might be returned to you during a subsequent receive request. You should ensure that your application is idempotent, so that receiving a message more than once does not cause issues.

This part is the most significant: The ReceiptHandle is associated with a specific instance of receiving a message. If you receive a message more than once, the ReceiptHandle is different each time you receive a message. It means that if the architecture allows several consumers of a single queue it is well possible that while one consumer is handling the message it was retrieved once more time by the other consumer and the ReceiptHandle that is going to delete the message is already different. Second significant thing is the request succeeds, but the message might not be deleted. SQS doesn't let you know that the messages were not deleted.

How the settings help

And why they help not in any case

The lower delays between retrievals the higher possibility of "undesired" retrieval of the message by another consumer during it's handling. But if consumers do long polling and cache ReceiptHandles the required VisibilityTimeout should be too long and with such big value, it is going to slow down the polling making it almost sequential.

Solution

Deletion of the handled messages should be done in a separate consumer in a case when you can't delete the message immediately after retrieval, otherwise, it is risking to be remaining as an orphan (a message that will never be handled and deleted again) in the queue.