AWS Lambda inside VPC. 504 Gateway Timeout (ENI?)

Question

I have a Serverless .net core web api lambda application deployed on AWS.

I have this sitting inside a VPC as I access ElasticSearch service inside that same VPC.

I have two API microservices that connect to the Elasticsearch service.

After a period of non use (4 hours, 6 hours, 18 hours - I'm not sure exactly but seems random), the function becomes unresponsive and I get a 504 gateway timeout error, "Endpoint cannot be found"

I read somewhere that if "idle" for too long, the ENI is released back into the AWS system and that triggering the Lambda again should start it up.

I can't seem to "wake" up the function by calling it as it keeps timing out with the above error (I have also increased the timeouts from default).

Here's the kicker - If I make any changes to the specific lambda function, and save those changes (this includes something as simple as changing the timeout value) - My API calls (BOTH of them, even though different lambdas) start working again like it has "kicked" back in. Obviously the changes do this, but why?

Obviously I don't want timeouts in a production environment regardless of how much, OR how little the lambda or API call is used.

I need a bulletproof solution to this. Surely it's a config issue of some description but I'm just not sure where to look.

I have altered Route tables, public/private subnets, CIDR blocks, created internet gateways, NAT etc. for the VPC. This all works, but these two lambdas, that require VPC access, keeps falling "asleep" somehow.

Answer 1

The is because of Cold Start of Lambda. There is a new feature which was release in reInvent 2019, where in there is a provisioned concurrency for lambda (don't get confused with reserved concurrency).

Ensure the provisioned concurrency to minimum 1 (or the amount of requests to be served in parallel) to have lambda warm always and serve requests

Ref: https://aws.amazon.com/blogs/aws/new-provisioned-concurrency-for-lambda-functions/

To get more context, Lambda in VPC uses hyperplane ENI and functions in the same account that share the same security group:subnet pairing use the same network interfaces.

If Lambda functions in an account go idle for sometime (typically no usage for 40 mins across all functions using that ENI, as I got this time info from AWS support), the service will reclaim the unused Hyperplane resources and so very infrequently invoked functions may still see longer cold-start times.

Ref: https://aws.amazon.com/blogs/compute/announcing-improved-vpc-networking-for-aws-lambda-functions/