Read SQS queue from AWS Lambda

13
votes

I have the following infrastructure:

I have an EC2 instance with a NodeJS+Express process listening on a port for messages (process 1). Every time the process receives a message it sends it to an SQS queue. Then I have another process in the same machine reading the queue using long polling (process 2). When it finds a message in the queue it inserts the data in a MariaDB database sitting on an RDS instance.

(Just to clarify, messages are generated by users, they send a chunk of data which can contain arbitrary information to the endpoint where the process 1 is listening)

Now I want to put the process that reads the SQS (process 2) in a Lambda function so that the process that writes to the queue and the one that reads from the queue are completely independent. The problem is that I don't know if this is possible.

I know that Lambda function are invoked in response to an event, and the events supported at the moment are S3, SNS, SES, DynamoDB, Kinesis, Cognito, CloudWatch and Cloudformation but NOT SQS.

I was thinking in using SNS notifications to invoke the Lambda function so that every time a message is pushed to the queue, an SNS notification is fired and invokes the Lambda function but after playing a bit with it I've realised that is not possible to create an SNS notification from SQS, it's only possible to write SNS notifications to the queue.

Right now I'm a bit stuck because I don't know how to continue. I have the feeling that is not possible to create this infrastructure due to the current limitations in the AWS services. Is there another way to do what I want or am I in a dead-end?

Just to extend my question with some research I've made, this github repo shows how to read an SQS queu from a Lambda function but the lambda function works only if is fired from the command line:

https://github.com/robinjmurphy/sqs-to-lambda

In the readme, the author mentions the following:

Update: Lambda now supports SNS notifications as an event source, which makes this hack entirely unneccessary for SNS notifcations. You might still find it useful if you like the idea of using a Lambda function to process jobs on an SQS queue.

But I think this doesn't solve my problem, an SNS notification can invoke the Lambda function but I don't see how I can create a notification when a message is received in the SQS queue.

Thanks

5
amazon-web-servicesamazon-ec2amazon-sqsaws-lambda
2 things which you can use to get your way out (1) Lambda can listen to SNS. If thats not what you want then (2) Make SQS Queue one of the subscribers of the SNS Topic [ Every SNS Message would be written to a SQS Queue] – Naveen Vijay
I think that here is where I start to get confused. I don't want to write SNS Messages to a queue. The messages that go into the queue are generated by users (they post data to a URL, my nodejs thread processes the request, formats the data and sends it to the SQS queue). Then what I'd like to do is, every time a user message is inserted into the queue, trigger somehow an SNS notification to invoke the Lambda function (which in fact, is made by making the Lambda function listen to SNS) – mIwE
Instead of connecting the dots between Lambda, SQS & SNS. I would like to recommend considering scheduling the lambda function to look at the queue, process the items if they exists. The other variant is using 2 Lambda functions - One will pull the read the items from SQS[scheduled] and push the items to SNS which then be handled by another processing Lambda function. – Naveen Vijay
That makes sense. In fact this solution is similar to the system we currently have using cronjobs and checking the queue every couple of minutes or seconds. I was also thinking in using CloudWatch to check the queue stats and trigger the lambda function if there are any messages. Thank you for the suggestion. I'm going to investigate further. – mIwE
glad that helped. I will write that as an answer. – Naveen Vijay

5 Answers

10
votes

There are couple of Strategies which can be used to connect the dots, (A)Synchronously or Run-Sleep-Run to keep the data process flow between SNS, SQS, Lambda.

Strategy 1 : Have a Lambda function listen to SNS and process it in real time [Please note that an SQS Queue can subscribe to an SNS Topic - which would may be helpful for logging / auditing / retry handling]

Strategy 2 : Given that you are getting data sourced to SQS Queue. You can try with 2 Lambda Functions [Feeder & Worker].

Feeder would be scheduled lambda function whose job is to take items from SQS (if any) and push it as an SNS topic (and continue doing it forever)

Worker would be linked to listen the SNS topic which would do the actual data processing

3
votes

We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.

enter image description here

Further details: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/

1
votes

I had a similar situation (and now have a working solution deploed). I have addressed it in a following manner:

enter image description here

i.e. publishing events to SNS; which then get fanned-out to Lambda and SQS.

NOTE: This is not applicable to the events that have to be processed in a certain order.

That there are some gotchas (w/ possible solutions) such as:

  • racing condition: lambda might get invoked before messages is deposited into the queue
  • distributed nature of SQS queue may lead to returning no messages even though there is a message note1.

The solution to both cases would be to do long-polling of SQS queue; but this does make your lambda bill more expensive.

note1

Short poll is the default behavior where a weighted random set of machines is sampled on a ReceiveMessage call. This means only the messages on the sampled machines are returned. If the number of messages in the queue is small (less than 1000), it is likely you will get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages in a particular ReceiveMessage response; in which case you should repeat the request. http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

1
votes

AWS SQS is one of the oldest products of Amazon, which only supported polling (long and short) up until June 2018. As mentioned in this answer, AWS SQS now supports the feature of triggering lambda functions on new message arrival in SQS. A complete tutorial for this is provided in this document.

I used to tackle this problem using different mechanisms, and given below are some approaches you can use.

  1. You can develop a simple polling application in Lambda, and use AWS CloudWatch to invoke it every 5 mins or so. You can make this near real-time by using CloudWatch events to invoke lambda with short downtimes. Use this tutorial or this tutorial for this purpose. (This could cost more on Lambdas)

  2. You can consider that SQS is redundant if you don't need to persist the messages nor guarantee the order of delivery. You can use AWS SNS (Simple Notification Service) to directly invoke a lambda function and do whatever the processing required. Use this tutorial for this purpose. This will happen in real-time. But the main drawback is the number of lambdas that can be initiated per region at a given time. Please read this and understand the limitation before following this approach. Nevertheless AWS SNS Guarantees the order of delivery. Also SNS can directly call an HTTP endpoint and store the message in your DB.

0
votes

We had some similar requirements so we ended up building a library and open sourcing it to help with SQS to Lambda async. I'm not sure if this fills your particular set of requirements, but thought it might be worth a look: https://read.iopipe.com/sqs-lambda-teaming-up-92c4096be49c