Serverless AWS Lambda@Edge: How to debug

2
votes

I have a python Lambda@Edge function deployed with serverless which is working OK using the test feature:

enter image description here

FYI it's setup as viewer-request:

enter image description here

functions:
  cfLambda:
    handler: handler.lambda_handler
    events:
      - cloudFront:
          eventType: viewer-request

And if I go to CloudWatch, I can see the logs:

enter image description here

Now when I test with cURL it fails with a 503:

HTTP/2 503 
content-type: text/html
content-length: 1019
server: CloudFront
date: Mon, 05 Apr 2021 07:24:45 GMT
x-cache: LambdaExecutionError from cloudfront
via: 1.1 XXXXXXXXXXXXXXXXXX.cloudfront.net (CloudFront)
x-amz-cf-pop: AMS50-C1
x-amz-cf-id: 4vYpBnOGd6yfgowoSpiCyBkh5cbV1g3IJf1H2Eheln89MpEnScL-1g==

However this time I get no logs in CloudWatch. Q1: How can I have traces of my Lambda@Edge CloudFront calls visible in CloudWatch?

If I read the Lambda@Edge debug guide it says that 503 status code is either:

  • too many executions (not my case, there is 0 traffic to that cloudfront, it's purely test)
  • function exceeded the Lambda function timeout quota

If I look at the console tests, they only consume ~220ms and ~75MB so I think we're way below the 5-second / 128MB limit for viewer request

enter image description here

If I look at the CloudFront logs they seem useless as they just confirm the 503:

E2HX7F6YEZN897.2021-04-04-16.a77a21e1:2021-04-04    16:34:12    SEA19-C3    389 35.247.33.169   HEAD    XXXXXXXX.cloudfront.net /   503 -   Mozilla/5.0%20(Windows%20NT%205.1)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/41.0.2224.3%20Safari/537.36  -   -   LambdaExecutionError    AcM5SX3ggB53fmjXO83xND_Lw3-eHXd8dlIZGEO53XaDMjuctRw==   example.org https   223 0.021   -   TLSv1.3 TLS_AES_128_GCM_SHA256  LambdaExecutionError    HTTP/1.1    -   -   51810   0.021   LambdaExecutionError    text/html   1019    -   -

Q2: Is there a way to increase the verbosity of the CloudFront logs (I couldn't find one)?

Q3: If I can't get my CloudFront Lambda@Edge calls in CloudWatch (Q1=no) and I can't increase verbosity of CloudFront Logs (Q2=no), how can I debug this further?

3
amazon-web-servicesaws-lambdaamazon-cloudfrontserverlessaws-lambda-edge
Have you tried to follow any recommendations on how to debug lambda@edge? For example, adding logging statements? – Marcin
The errors were already being logged, I was just looking at the wrong place. Going to the CloudWatch /aws/lambda/us-east-1.<Your-function-name> logs allowed me to find the errors of the lambda function when executed via edge – Max
Thanks for letting me know. Glad it worked out :-) – Marcin

3 Answers

4
votes

  • The Lambda@Edge function must be deployed to the us-east-1 region.

  • The x-amz-cf-pop header gives a hint about where the request was executed. You can refer to this unofficial list here.

  • For us-east-1 the logs can be found in CloudWatch under the group /aws/lambda/<Your-function-name>. For any other region the log group would be /aws/lambda/us-east-1.<Your-function-name>. If you know the region, then select the appropriate region. Go to CloudWatch and search the appropriate log group. enter image description here

  • You can also navigate to the appropriate logs from the CloudFront page. Go to the Monitoring section -> Choose your Distribution -> View Distribution Metrics -> Lambda@Edge Errors. The graph would display the errors from all the regions when you hover over the data points. Once you know the region where the error is happening, you can select the same followed by the Lambda function and finally click on View logs. Refer to the below image. enter image description here

  • Take a look at some official examples here. For the viewer-request event, the ones manipulating the request are relevant.

  • The CloudFront logs can be accessed at /aws/cloudfront/LambdaEdge/<YourDistributionId>

Hopefully this helps you proceed further.

1
votes

There's a few key considerations here.

Firstly, it's important to note how edge functions work.

Once deployed, your edge function is replicated across all AWS regions.

When a Cloudfront request comes in, it gets routed to the nearest available region to the user's location.

As a result, the function executes its logs in the region closest to the caller and not in the region of initial deployment.

Sometimes this may be a little bit counter-intuitive. It may be possible to be based in England, but be closer to the Ireland region for example.

Next, it's possible your lambda functions don't have the required permissions to log to Cloudfront.

When using Cloudfront logs you need to explicitly give your function logging permissions.

Consider the following IAM role:

{
"Version": "2012-10-17",
"Statement": [
    {
        "Effect": "Allow",
        "Action": "logs:CreateLogGroup",
        "Resource": "arn:aws:logs:*:*:*"
    },
    {
        "Effect": "Allow",
        "Action": [
            "logs:CreateLogStream",
            "logs:PutLogEvents"
        ],
        "Resource": [
            "arn:aws:logs:*:*:log-group:*:*"
        ]
    }
  ]
}

This role enables the function to create log groups, log streams and to put events into the logs for both Cloudfront and Cloudwatch.

Notice the wildcards, this is because when deploying to the edge you need to provide Cloudfront specific permission using the arn arn:aws:logs:*:*:log-group:/aws/cloudfront/*.

-1
votes

Have you tried AWS x-ray? https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html

It's a distributed tracing util lets you debug lambda functions. Probably will require some setup overhead, but in the long run might come in handy