Tracking a message on multiple AWS SQS queues

0
votes

I have around 3 AWS Lambda functions taking the following form:

  • Lambda function 1: Reads from an SQS queue and puts a Message on an SQS queue (the incoming and outgoing message formats are different)
  • Lambda function 2: Reads the message from Lambda function 1, and puts a Message on an SQS queue (the incoming and outgoing message formats are different)
  • Lambda function 3: Reads the message from Lambda function 3, and updates storage.

There are 3 queues involved and the message format (structure) in each queue is different, however they have one uniqueId which are the same and can be used the relate one to each other. So much question is, is there any way in SQS on or some other tool to track the messages, what I'm specifically looking at is stuff like:

  • Time the message was entered into the queue
  • Time the message was taken by the Lambda function for processing

My problem is that the 3 Lambda functions individually perform within a couple of milliseconds, but the time taken for end to end execution is way too long, I suspect that the messages are taking too long in transit.

I'm open for any other ideas on optimisation.

1
amazon-web-servicesaws-lambdaamazon-sqs
I think sqs and x-ray could be helpful. Probably would have to add annotations to the x-ray segments with your uniqueId so that you can later filter out messages you want. - Marcin
Thanks, I did see this, but this means I need to change my code now right to include these additional headers? - pandith padaya
Yes. I don't have entire picture of this in my head, but this is the first thing I would consider. Alternatively, you could just add a timestime to each message in each lambda, and manually check how long it takes from the first to last lambda. - Marcin
And how long is "way too long"? Maybe there is some other cause for this, e.g. you are delaying your messages or using delay queues? - Marcin

1 Answers

3
votes

AWS Step Functions is specifically designed for passing information between Lambda functions and orchestrating the whole process.

However, you would need to change the way you have written your functions to take advantage of Step Functions.

Since your only real desire is to explore why it takes "way too long", then AWS X-Ray should be a good way to gather this information. It can track a single transaction end-to-end through each process. I think it's just a matter of including a library and activating X-Ray.

See: AWS Lambda and AWS X-Ray - AWS X-Ray

Or, just start doing some manual investigations in the log files. The log shows how long each function takes to run, so you should be able to identify whether the time taken is within a particular Lambda function, or whether the time is spent between functions, waiting for them to trigger.