AWS SQS doesn't reliably trigger Lambda

5
votes

I've set up a small serverless app using Lambda and SQS. In my case i wanted to trigger a lambda every time a message is added to a SQS Queue.

functions in my serverless.yml

functions:
  collectGame:
    handler: js/collect.collectGame
    memorySize: 128
    timeout: 10
    events:
    - sqs:
        arn:
          Fn::GetAtt:
            - gameRequestQueue
            - Arn
    - http:
        method: post
        cors:
          origin: "https://my-api-url.com"
        path: get/game/{id}
        private: true
        request:
          parameters:
            paths:
              id:true

I tested the process by sending 31 Messages at once to the Queue but realized that only 9 Lambdas get executed (by looking into the cloudwatch logs). I looked into the Queue and can confirm that its being filled with all the messages and that its empty after the 9 Lambdas have been triggered.

I'd expect to have 31 Lambda executions but thats not the case. Anyone knows potential reasons why my Lambdas are not being triggered by the messages?

3
amazon-web-servicesaws-lambdaamazon-sqsserverless
Does your lambda get invoked with multiple messages? – cementblocks
Are you sure there were only 9 executions and not just 9 log groups in cloud watch? – Artem Arkhipov
@ArtemArkhipov I tried it again and now i see 11 Invocations inside a total of 8 log groups – Lucca

3 Answers

10
votes

Your lambda function is probably being invoked with multiple messages. You should be able to set the BatchSize to 1 when you create the event source mapping, if you only want one message to be sent per lambda invocation

It looks like you are using the serverless framework. See their SQS event documentation for setting the batch size.

2
votes

For anyone using aws sam here is the link that mentions batch size: here, look for the subheading 'Configuring a Queue as an Event Source'. And here is the code that works for me to set this up in the yaml together with a DLQ:

  # add an event trigger in the properties section of your function
  Events:
    MySQSEvent:
      Type: SQS
      Properties:
        Queue: !GetAtt MySqsQueueName.Arn              
        BatchSize: 1

# then define the queue
  MySqsQueueName:
    Type: AWS::SQS::Queue  
    Properties: 
      VisibilityTimeout: 800
      ReceiveMessageWaitTimeSeconds: 10
      DelaySeconds: 10
      RedrivePolicy:
        deadLetterTargetArn: !GetAtt MyDLQueue.Arn
        maxReceiveCount: 2

  # define a dead letter queue to handle bad messages
  MyDLQueue:
    Type: AWS::SQS::Queue  
    Properties: 
      VisibilityTimeout: 900

Hope this helps someone - this took me ages to work out for my app!

1
votes

i was also facing the exact same issue. The problem was in my lambda function. If the batch size is more than 1, in that case in a single lambda invocation, multiple SQS messages will be passed to lambda (based on batch size), just handle all the messages in lambda (by iterating through all the messages).

check your event Records array for multiple messages.

{Records: [{..},{..},{..}]}