AWS Lambda to read from SQS queue

1
votes

I have an AWS Lambda function to read from an SQS queue. The lambda logic is basically to read off one message from SQS and then it processes and deletes the message. Code to read the message being something like.

 ReceiveMessageRequest messageRequest =
        new ReceiveMessageRequest(queueUrl).withWaitTimeSeconds(5).withMaxNumberOfMessages(1);

Now my question is what is the best way to trigger this lambda and how does this lambda scale for instance, if there are let's say 1000 messages in the queue so will there be a 1000 lambdas running together, since in my case one lambda can read only one message off the queue.

Any pointers on best practices around this kind of design.

4
amazon-web-servicesaws-lambdaamazon-sqsserverless
Similar to stackoverflow.com/questions/48882830/… – Renato Gama
Why are you wanting to trigger Lambda based on an SQS message? SQS is for buffering information, but Lambda can operate in real-time. Is there some way you can change the source system so that it triggers Lambda or SNS instead of SQS? – John Rotenstein

4 Answers

1
votes

Right now you best option is probably to setup an AWS Cloudwatch event rule that calls the lambda function on the interval that you need.

Here is a sample app from AWS to do just that:

https://github.com/awslabs/aws-serverless-sqs-event-source

I do believe that AWS will eventually support SQS as a event type for AWS lambda, which should make this even easier, but for now you best choice is probably a version of the code I linked above.

1
votes

We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.

enter image description here

Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

-1
votes

There are probably a few ways to do this, but I found this guide to be fairly helpful when I tried to implement the same sort of functionality you are describing in Node.js. One downside to this strategy is that you can only poll the queue every 60s.

The basic workflow would look something like this:

  • Set up a CloudWatch Alarm that gets triggered when the queue has a certain number of messages.

  • The Cloudwatch alarm then posts to SNS

  • The SNS message triggers a Lambda scale() function

  • The scale() function updates a configuration record in a DynamoDB table that sets the number of worker processes needed

  • You then have a main CloudWatch Schedule that invokes a worker() function every 60s

  • The worker() function reads configuration from DynamoDB to determine how many concurrent processes are needed, based on the queue size.

  • Worker() then invokes the appropriate number of process() functions

  • Process() function consumes messages from SQS, performs your main application logic, and then removes the item from the queue.

You can find an example of what the scaling functions would look like in Node.js here

I have used this solution in a production environment for almost a year without any issues, even with thousands of messages in the queue. If you cut out the scaling portion it is only going to do one message a time.