Random status code: 502 errors on AWS api gateway connected to lambda

Question

I have multiple lambdas exposed with api gateway using proxy integration. From time to time i'm getting strange errors with status code 502. There is nothing in lambda cloud watch logs. Below i posted api gateway logs for sample request:

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Received response. Integration latency: 231 ms

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Endpoint response body before transformations:
{
    "Message": "An error occurred and the request cannot be processed.",
    "Type": "Service"
}

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Endpoint response headers: 
{
    Connection=keep-alive, 
    x-amzn-RequestId=0cbc9dee-f1bd-11e7-857b-91f7f814692c, 
    x-amzn-ErrorType=ServiceException, 
    Content-Length=86, 
    Date=Fri, 05 Jan 2018 02:06:32 GMT, 
    Content-Type=application/json
}

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Execution failed due to configuration error: Malformed Lambda proxy response

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Method completed with status: 502

Basically it seems that api gateway cannot reach lambda and call to lambda is returning:

(0cbbd9f5-f1bd-11e7-92c0-4d5d3b7d0380) Endpoint response body before transformations:
{
    "Message": "An error occurred and the request cannot be processed.",
    "Type": "Service"
}

Is there any one else experiencing those issues? Only possible fix from my side is to write retry mechanism but from my side it looks rather that i am missing some configuration or it's AWS failure which they should handle.

Usually the AWS API gateway returns HTTP 502 (Bad Gateway) when an exception is not handled by the function(proxy mode). There's a message in the log: "Execution failed due to configuration error: Malformed Lambda proxy response", that means that for some reason your Lambda function didn't return the response in the expected format. Try to log the entire execution of your lambda functions to find out whats wrong. — Tom Melo
@TomMelo Thanks for your response Tom! As i wrote above call to lambda returns: " Endpoint response body before transformations: { "Message": "An error occurred and the request cannot be processed.", "Type": "Service" }" Which is later mapped to "Execution failed due to configuration error: Malformed Lambda proxy response by api gateway". I have entire lambda function surrounded by try/catch block so there is no way it comes from my code. What's more aws cloud watch is empty for that request (no start/finished logs as usual) so it doesn't even reach aws lambda. — Pawel
That response is from Lambda to API Gateway. The recommendation is to retry any 5xx errors from the client side to improve reliability. Your best bet to resolve this issue is to open a support ticket with AWS. — Abhigna Nagaraja
Already did that. No response so far forums.aws.amazon.com/thread.jspa?messageID=719917 i don't have commercial support plan so aws forum is all i can try. — Pawel
I've seen the same behavior in a number of instances over the past 1 year + for our production apps. It's totally random. It seems like API Gateway didn't get any response from the Lambda (or not the response it was expecting), totally at random, and barfs. But after a matter of seconds to up to a minute, it will recover and pretend everything's fine. — Joshua

raevilman raevilman · Accepted Answer · 2018-05-14T06:31:15

I'm listing here one possible reason...

When an AWS Lambda is configured to run in VPC. It takes one IP per execution from VPC.

And if VPC doesn't much free IPs then your lambda will fail silently :(

I've personally faced issues in regards to limited IP, increasing the IPs solved the issue.

Below text from this link

The subnets you specify should have sufficient available IP addresses to match the number of ENIs.

We also recommend that you specify at least one subnet in each Availability Zone in your Lambda function configuration. By specifying subnets in each of the Availability Zones, your Lambda function can run in another Availability Zone if one goes down or runs out of IP addresses.

Note

If your VPC does not have sufficient ENIs or subnet IPs, your Lambda function will not scale as requests increase, and you will see an increase in function failures. AWS Lambda currently does not log errors to CloudWatch Logs that are caused by insufficient ENIs or IP addresses. If you see an increase in errors without corresponding CloudWatch Logs, you can invoke the Lambda function synchronously to get the error responses (for example, test your Lambda function in the AWS Lambda console because the console invokes your Lambda function synchronously and displays errors).

Random status code: 502 errors on AWS api gateway connected to lambda

1 Answers