Gunicorn worker terminated with signal 9

18
votes

I am running a Flask application and hosting it on Kubernetes from a Docker container. Gunicorn is managing workers that reply to API requests.

The following warning message is a regular occurrence, and it seems like requests are being canceled for some reason. On Kubernetes, the pod is showing no odd behavior or restarts and stays within 80% of its memory and CPU limits.

[2021-03-31 16:30:31 +0200] [1] [WARNING] Worker with pid 26 was terminated due to signal 9

How can we find out why these workers are killed?

2
pythonflaskgunicorn
Did you manage to find out why? Having the same issue, and tried specifying the --shm-size - but no avail. - lionbigcat
Our problems seem to have gone away since we started using --worker-class gevent. I suspect Simon is right and this was either an out of memory error, or a background process running for too long and the main process (1) decided to kill it. - Jodiug
Meta: I'm not sure why this question is being downvoted. Please drop a comment if you feel it needs further clarification. - Jodiug
I have the same problem, and gevent did not solve it. does anyone knows why this started all of a sudden? was there a change in gunicorn or in kube? - Blop
also related to a non answered question: stackoverflow.com/questions/57745100/… - Blop

2 Answers

22
votes

I encountered the same warning message.

[WARNING] Worker with pid 71 was terminated due to signal 9

I came across this faq, which says that "A common cause of SIGKILL is when OOM killer terminates a process due to low memory condition."

I used dmesg realized that indeed it was killed because it was running out of memory.

Out of memory: Killed process 776660 (gunicorn)
0
votes

In my case the problem was in long application startup caused by ml model warm-up (over 3s)