SQS Lambda serverless: the Sqs Message is not consumed

Question

I deployed a Python Post API lambda, using Serverless framework.

Detail of code as below:

app.py

from flask import Flask, jsonify, request

app = Flask(__name__)

@app.route('/process', methods=['POST'])
def process():
    content = request.json
    print content
    return jsonify(content)

if __name__ == "__main__":
    app.run()

serverless.yml

service: my-service
plugins:
  - serverless-python-requirements
  - serverless-wsgi
custom:
  wsgi:
    app: app.app
    packRequirements: false
  pythonRequirements:
    dockerizePip: false
package:
  exclude:
    - node_modules/**
    - venv/**
provider:
  name: aws
  runtime: python2.7
  stage: dev
  region: eu-west-1

functions:
  app:
    handler: wsgi.handler
    events:
      - http: ANY /
      - http: 'ANY {proxy+}'

I configured the SQS to trigger the lambda. I send a SQS Message, the lambda is triggered (I checked it from CloudWatch), but the SQS message is stuck in flight. It seems that SQS cannot be consumed by lambda.

Any suggestion is appreciated

****UPDATE*****

After check the CloudWatch carefully, I found some exceptions

u'headers': KeyError
Traceback (most recent call last):
File "/var/task/wsgi.py", line 100, in handler
return serverless_wsgi.handle_request(wsgi_app, event, context)
File "/var/task/serverless_wsgi.py", line 73, in handle_request
headers = Headers(event[u"headers"])
KeyError: u'headers'

Is the Lambda function exiting cleanly, not throwing any exceptions? — Michael - sqlbot
Because I only print the SQS message (refer to my code), so in CloudWatch, I only see the SQS message, not any exception. I also had the thought of lambda is not finished, but how to exit lambda properly? I did return in lambda. — Phong Vu
Not sure if this is true for all languages (I almost exclusively use Node.js in Lambda) but you should see at least three log entries for each invocation -- stop, start, and the output from your print. Not sure why you wrote this with flask, since it's behind SQS instead of API Gateway. I may be wrong, but this doesn't seem right. — Michael - sqlbot
I am planing to move a worker (python) to lambda, so that I try to write lambda in Python. Like you, I wrote most lambda so far in nodejs, and using Serverless to deploy it. As AWS documentation, SQS trigger lambda which actually handle post request with sqs message as body. — Phong Vu

vfcoelho182 vfcoelho182 · Accepted Answer · 2020-05-01T07:00:36

Looking at your logs, it seems that Flask's WSGI is expecting a headers node within the payload you're receiving, which would be there should your event be an HTTP event. That's not the case when you wire two AWS services directly. The event object will be different for each service. Since you are not handling HTTP requests, I would suggest you to remove Flask from this lambda.

Your code then would look like this:


import json

def process(event,context):
    content = request.json
    print(event)
    return json.dumps(content,ensure_ascii=False)

service: my-service
plugins:
  - serverless-python-requirements
package:
  exclude:
    - node_modules/**
    - venv/**
provider:
  name: aws
  runtime: python2.7
  stage: dev
  region: eu-west-1

functions:
  app:
    handler: app.process
    events:
      - sqs:
          arn: arn:partition:service:region:account-id:resource-id
          batchSize: 1

Your SQS queue ARN is displayed on your queue details on AWS Console.

Here is a complimentary read:

SLS and SQS Queues

SQS Lambda serverless: the Sqs Message is not consumed

1 Answers