aws lambda + SQS - batch messaging with concurrency as one

Question

I have set a sqs trigger in lambda with

batch size : 3

batch window : 300 seconds

concurrency: 1

The SQS queue is set with

visiblity timeout: 3 minutes

The idea here is to process 3 files at a time.

This is how the lambda code looks like

def lambda_handler(event, context):
    
    maximum_jobs = 3
    sqs_client = boto3.client(
        'sqs'
    )
    
    
    for i in range(len(event["Records"])):
        msg_string = event["Records"][i]["body"]
        
        
        if get_active_executions() < maximum_jobs:

        start_execution()
                
        receipt_handle = event["Records"][i]["receiptHandle"]
        
        delete_sqs_message(sqs_client, "myqueue",  receipt_handle)
   
        
        else:
            print(f"Already {maximum_jobs}  jobs running")
        
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

To test the success scenario, i pushed 6 entries into the queue.

First 3 got processed immediately and first 3 messages are deleted from the queue.
Other 3 were waiting in flight mode (since, i have set the concurrency as 1 and batch size as 3)
After visibility timeout of 3 mins, the remaining 3 messages came back to the queue and the trigger picked up the remaining 3 files. the remaining 3 also got deleted.
All 6 files got processed successfully.

Now, I tested the same code with 9 files

First 3 got processed immediately (say file1, file2, file3)
The second batch (after 3 mins) got triggered (say file4, file5, file6) . However at that time, the first batch was running. I could see the log - "Already 3 jobs running". Hence, it didnt get process. As per the code, i didnt delete it either.
Now, the third batch got triggered (say file7, file8, file9). By this time, the first batch execution was over and the hence all 3 got processed and deleted from queue successfully.

I was expecting for another batch trigger in 3 mins (file4, file5, file6). However, it didnt happen. I noticed that the queue is empty.

Does the message disappear from the queue if the event is triggered and not manually deleted? what is the expected behaviour. Am i missing something in the code ?

What's happening in start_execution() ? Are you triggering another task outside of your Lambda function? If so, do you need to wait for that? It sounds a bit like getActiveExecutions() is checking some outside task if something is running and this task takes longer than expected, hence you're seeing this Already 3 jobs running because you didn't wait for the task to complete in your first batch. — s.hesse
The other problem: if SQS triggers your Lambda with one or more messages and your Lambda function does not throw an error, SQS will automatically delete the messages from the queue. I.e. you don't need to remove them in your code, it's done automatically. See also here: docs.aws.amazon.com/lambda/latest/dg/with-sqs.html : When your function successfully processes a batch, Lambda deletes its messages from the queue — s.hesse
Oh!!. The second batch didnt threw error. hence the messages got deleted from the queues by lambda. How, should i deal with this scenario ? i want them to be back in queues. Should i need to re-insert again or throw an error in lambda, so that it wont get deleted from queue. — srinath
Not sure about your use case and about the effect of your processing order but throwing an exception is fine here but maybe not ideal because you’d repeat throwing an exception until your tasks finish -> unnecessary costs in my opinion. Maybe you find a better way? — s.hesse

Willem Willem · Accepted Answer · 2021-02-16T13:30:38

Does the message disappear from the queue if the event is triggered and not manually deleted?

When the lambda successfully processes a batch, all messages in the batch are removed from the queue.

It is OK to delete the message inside the lambda function, but you typically do not need to.

If the lambda throws an exception, the messages (that were not deleted by the lambda) will "return" to the visible queue after the visibility timeout.

It is usually better to design a solution that handles exceptions gracefully and not throw an exception. But it depends on the detail of your problem.

aws lambda + SQS - batch messaging with concurrency as one

1 Answers