AWS Python lambda times out forever after first timeout

Question

Problem:

I have a python lambda that constantly receives data every second and puts it into DynamoDB. I noticed that after the first time DynamoDB takes a little more and the function times out, all the following calls also timeout and it never recovers. The way to bring the lambda back to normal is to redeploy it. When it starts timing out, it does not display any logs. It times out without executing any of the code.

Below is a picture of our console that represents the issue.

In order to reproduce the issue faster with this function I did the following:

Redeploy it and see it is working fine.
Reduce the memory available to the lambda to the minimum and timeout to 1 second. This will cause the first timeout
Increase back the memory of the lambda to normal and even increase the timeout. However, the timeouts persist

Is there a way to resolve this issue without having to redeploy?

I have seen the same description of issue but with nodejs in this post: https://forums.aws.amazon.com/thread.jspa?threadID=234417.

I haven't seen any description related with the python lambda env

More information about the setup:

Lambda environments tested: python3.6 and python3.7

Tool to deploy lambda: serverless 1.57.0

serverless plugins used: serverless-python-requirements, serverless-wsgi

I am not using any VPC for the lambda

Thank you for the help,

What is the Lambda function doing? Can you share the code? Are there any errors or other information in the Lambda function's logs that are relevant to your question? — Mark B
I can't share the code, but it is fairly simple. It is a small lambda that puts data into dynamoDB. I have seen people complaining about the same issue with nodejs here: forums.aws.amazon.com/thread.jspa?threadID=234417. But I haven't seen anything related with Python. There are no logs when it times out. It doesn't even start the normal code. — Pedro Borges
Please try to share a small example, removing all sensitive parts, that exhibits this problem. The WSGI part of this is unusual so it would be hard for those of us who don't use WSGI with serverless Python to attempt to repro. — jarmod
Will try to isolate it and put in a repository so others can access — Pedro Borges

Pedro Borges Pedro Borges · Accepted Answer · 2019-11-07T16:14:21

Figured out the trigger for the bug. When the lambda function zip uploaded is too large, after the first time it times out, it never recovers!

My solution was to carefully strip out the unnecessary dependencies to make the package smaller.

I created a repository using a docker container for people to reproduce the issue more easily:

https://github.com/pedrohbtp/bug-aws-lambda-infinite-timeout

Thanks for the messages in the comments. I appreciate whoever takes time to try to help here in SO.

AWS Python lambda times out forever after first timeout

1 Answers