AWS Lambda w/ SQS trigger, SQS Lambda Destinations, never adds to destination queue

Question

I have a simple lambda function that is triggered from a SQS queue and I'm using the new Lambda Destinations functionality.

It is set up to trigger from QUEUE_A, do some modification of the payload body, then send it to QUEUE_B on success, or QUEUE_ERRORS on failure.

QUEUE_B and QUEUE_ERRORS are set up as Destinations on the lambda function.

When I trigger the lambda from the CLI, I get a record on QUEUE_B with a good record, and on QUEUE_ERRORS on a bad record. So, it seems to be working.

But, when the lambda is triggered from SQS, I never get a record on QUEUE_B or QUEUE_ERRORS. A good record runs the lambda, and on a bad record it goes to QUEUE_A_DEADLETTER, which I do not want.

I have tried configuring QUEUE_A to have no retry/deadletter - if I do that, on a bad record it will just retry forever (no matter how low I set the visibility/retries).

What can I try next?

EDIT:
CloudWatch shows me exactly what I expect to see - I see good logs on a "good" record, and stack traces/exceptions on "bad" records, so it is not a problem within the function itself AFAIK.

EDIT: Replacing the SQS triggers and destinations with SNS triggers and destinations are working. So, I think this is related to SQS being sync and SNS being async? Does anybody know?

Sounds like you are getting errors within the Lambda function. Have you looked at CloudWatch Logs to see the log output of the Lambda function? You could also add some additional debug to see whether the SendMessage() call succeeded. — John Rotenstein
CloudWatch shows good logs on my "good" records, and exceptions on my "bad" records. Where is SendMessage() defined? I am not trying to push to SQS in the function code, just as a Destination. I also expect a "bad" record to go to QUEUE_ERRORS but it doesn't. — willoller
Ah! It's using the new AWS Lambda Destinations functionality. Sorry, with that you won't need SendMessage(). — John Rotenstein

Sarthak Jain Sarthak Jain · Accepted Answer · 2020-01-10T12:24:10

Destinations will not be triggered if you invoke the Lambda function synchronously. You can read this article to get a better idea.

The primary use case of Destinations is to know about the async execution results of Lambda functions, primarily to get more visibility into the execution details like request and response contexts, payloads, exception stack traces etc. So if a Lambda was invoked synchronously (say, using cli or via an SQS trigger), no messages will be delivered to the Destinations endpoints.

When you used the CLI you would have used aws lambda invoke-async. Instead if you use aws lambda invoke (which executes Lambda synchronously), you will see the same problems, your destination endpoints will not receive the message.

You can keep your Destinations endpoints still as SQS (you will see a working example in the article above), but your Lambda trigger will have to change to an asynchronous one.

AWS Lambda w/ SQS trigger, SQS Lambda Destinations, never adds to destination queue

1 Answers