supervisord stopping child processes

59
votes

One of the problems, I face with supervisord is that when I have a command which in turn spawns another process, supervisord is not able to kill it.

For example I have a java process which when runs normally is like

 $ zkServer.sh start-foreground
 $ ps -eaf | grep zk
 user 30404 28280  0 09:21 pts/2    00:00:00 bash zkServer.sh start-foreground
 user 30413 30404 76 09:21 pts/2    00:00:10 java -Dzookeeper.something..something

The supervisord config file looks like:

[program:zookeeper]
command=zkServer.sh start-foreground
autorestart=true
stopsignal=KILL

These kind of processes which have multiple childs are not well handled by supervisord when it comes to stopping them from supervisorctl. So when I run this from the supervisord and try to stop it from supervisorctl, only the top level process gets killed but not the actual java process.

6
bashapache-zookeepersupervisordubuntu-11.04
It is my understanding that the systemd init-replacement's use of cgroups allows for reliably tracking child processes. It might be suitable for your needs. - sarnold

6 Answers

105
votes

The same problem was encountered by Rick Hanlon II here: https://coderwall.com/p/4tcw7w

Option stopasgroup=true should be set in the program section for supervisord to stop not only the parent process but also the child processes.

The example is given as:

[program:some_django]
 command=python manage.py runserver
 directory=/dir/to/app
 stopasgroup=true

Also, have in mind that you may have an older package of supervisord that does not have "stopasgroup" functionality. I tried these Debian packages on Raspberry Pi:

  • supervisor_3.0a8 does not work.
  • supervisor_3.0b2-1 works as expected.
12
votes

Doing the following early in the main bash script called by supervisord fixed the problem for me:

trap "kill -- -$$" EXIT

This kills the entire process group when the main script exits, such as when it is killed by supervisord.

6
votes

A feature was recently added to supervisord to send SIGKILL to the whole process group. It's in github but not officially released yet.

If the process id is available in a file, you can use the pid-proxy program

4
votes

The following article has an in-depth discussion of the problem:

http://veithen.github.io/2014/11/16/sigterm-propagation.html

4
votes

try this supervisor program config:

stopasgroup=true
killasgroup=true
stopsignal=INT
0
votes

You can also use priorities in /conf.d/your-configuration.conf file. For example, if you want to run zookeeper first and then kafka you can specify two programs.

Lower priority means that the program starts first and stops last.