How can I ensure that a downstream API called by Lambda integrated with SQS is called at least 2 times before message going to DLQ?

0
votes

I have lambda using SQS events as inputs. The SQS queue also has a DLQ. The lambda function invokes a downstream Restful API (call this operation DoPostToAPI()) I need to guarantee that the lambda function attempts to call DoPostToAPI() at least 2 times (before message goes to DLQ)

What configuration of Lambda Retries and SQS Redrive policy would I need to set in order to accomplish the above requirement?

I need to be 100% certain that messages that arrive on the DLQ only arrive because they have attempted to been sent to downstream API DoPostToAPI() 2 times, and that messages dont arrive in DLQ for any other reason, if possible.

To me, it makes sense that messages should only arrive on the DLQ if the operation was attempted, and not for other reasons (i.e. I dont want messages to arrive on DLQ purely because of throttling, since the DoPostToAPI() should be attempted first before sending to DLQ) Why would I want messages on DLQ if the lambda function operation wasnt even attempted? In order words, I need the lambda operation to be guaranteed to be invoked before item moves to DLQ.

Can I get some help on this? Is it possible to guarantee that messages on the DLQ have arrived because of failed DoPostToAPI() api calls? Or is it (more unfortunate) possible that messages arrive on DLQ for reasons other than failed calls to downstream API?

From what I have read online so far, its possible that lambda , after doing receive on SQS message and moving the message to invisibile on the queue, could run into throttling issues and re-attempt the lambda invocation. But if it runs into lambda throttling again, it could end up back on main queue, which if it reaches its max receive count, could place the message on the DLQ without the lambda having been attempted at all. Is this correct?

For simplicity lets imagine the following inputs

  • SQSQueue1
  • SQSQueue1DLQ
  • LambdaFunction1 --> ServiceClient1.DoPostToAPI()

What is the interplay between the lambda "maximum_retry_attempts" and the SQS redrive_policy "maxReceiveCount"

1
amazon-web-servicesaws-lambdaamazon-sqs
Lambda has no idea what was "attempted". It only knows how the Lambda finished/exited. Based on this it will put a message in the DLQ. If you want your code to retry then you need to code it that way. - stdunbar
Are you 100% sure about that? I mean, retries can happen by the queue redrive policy can’t they? Also, if “lambda” doesn’t know about retries, why then does it even have MaximumRetryAttempts property on the lambda itself? - jbooker
Perhaps it's semantics. The Lambda runtime environment has no idea what your Lambda does. It will retry if your Lambda throws an unhandled exception or times out, etc. Your code has to tell the Lambda environment that it failed. My comment related to that. If your code catches the remote API exception, logs it, and returns properly then the Lambda succeed from the perspective of the Lambda runtime. If your code instead throws an exception when you can't interact with the downstream service then you can use retries. I still think you'll need to code 2 retries internally though. - stdunbar
“Code 2 retries internally” what do you mean by this exactly? Also, I still don’t think the details surrounding the interplay between the lambda “retry attempts” and the SQS maxRecieveCount has been addressed. If you feel confident in your understanding, please feel free to provide answer , including the interplay. I believe there are still unanswered questions. It would be great to get a concise answer, for me and the broader community. - jbooker
Again, my goal is to guarantee the downstream API is called At Least Twice. This is a hard requirement that really needs emphasis. I thank you and everyone who can help ensure I can meet this requirement. - jbooker

1 Answers

0
votes

In order to ensure your lambda attempts retries when using SQS, You only need set the SQS property

maxReceiveCount

This value controls how many lambda invocations will be attempted for a given batch before a message goes to the Dead Letter queue.

Unfortunately, the lambda property

maximum_retry_attempts

Does not apply for lambda functions using SQS as function event trigger.