Resource clean up after killing storm topology

Question

We have a storm topology which interacts with a MariaDB database. Our Bolts implement the IRichBolt interface and override the lifecycle methods. We open a db connection in our prepare method and close it in the cleanup method. The cleanup method documentation says:

Called when an IBolt is going to be shutdown. There is no guarentee that cleanup will be called, because the supervisor kill -9's worker processes on the cluster. The one context where cleanup is guaranteed to be called is when a topology is killed when running Storm in local mode

And the kill -9 command kills the process without cleaning up any resources. So we have come to this conclusion that on killing the topology it is not necessary that the cleanup method would be called and the db connection will be closed.

So moving forward to my question, we have a shell script for topology deployment which when executed kills the current topology with a timeout of 0 and deploys a new topology. We are facing an issue at db level that there are many connections opened which gives us the hint that previous connections were not closed. (The one opened in the previous topology).

Is our assumption correct? Will increasing the timeout clean up all the resources?

Answer 1

No. Increasing the topology timeout won't affect how long your worker has to clean up. When you use e.g. a 30s timeout, that just shuts off the spout and gives the rest of the topology 30 secs to finish processing.

What you want is to increase how long Storm allows a worker to spend shutting down before force killing. The https://github.com/apache/storm/blob/b07413670fa62fec077c92cb78fc711c3bda820c/storm-server/src/main/java/org/apache/storm/DaemonConfig.java#L780 option allows you to specify how long Storm will wait before sending the kill -9. The default is 3 seconds. You need to set this option in storm.yaml, setting it in the topology configuration will have no effect.