SQS Logging for insertion and removal from queue

Question

I'm using Amazon SQS for my application in a producer/consumer context. I want to enable queue level logging where I can see items put on the queue and removed from it later. How can I do that?

I have read the following:

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/logging-using-cloudtrail.html

However, that doesn't suffice for my use case. Are we not allowed to do this with AWS queues?

Answer 1

What you're trying to achieve is not possible with just SQS. Possible solutions include:

1. Implement some middleware API between you producer and SQS queue. API level would log requests from producer.
2. Use Kinesis instead of SQS. Kinesis allows you to replay/analyze records created in last 24 hours.
3. Implement logging in consumer.
4. Use Lambda function that will (with help of CloudWatch Event Rule triggers) read SQS queue once a minute, log records and put them in another SQS queue for later processing by consumer.
5. Use different type of queue that allows logging. For example, Redis has MONITOR command for that.

Answer 2

In addition to Sergey Kovalev answer, one now has the option for Lambda functions to be triggered by SQS events.

You simply:

select the SQS queue you want as the event source for your Lambda function

Answer 3

I understand your pain. Even I had the issue where SQS was not behaving as expected and I was looking for logs to understand the problem.

SQS don't publish logs, all SQS APIs are synchronous so the client get the appropriate response.

The solutions mentioned above are the workarounds Among them having Loggin at produce and consumer might not help much. Because in my case I did had logging at produce and consumer, but still what exactly SQS ran into and when will not be visible.