How to add a SIGUSR1 handler to a Twisted program?

Question

I am looking for a way to inspect the state of my Twisted-based program, so I can determine the number of connected clients and have a look at other metrics I collect (such as the time of connection establishment or the time of last contact).

My thought was to add a signal handler for SIGUSR1 to the process, such that when the process receives it, it will dump the state to a file in a known location. There are several problems with this:

Twisted overrides the signal handler with its own handler, because when the signal arrives the process ends with "user signal 1 received" on stdout and my code is not invoked
archaeological research on a mailing list revealed that Twisted will not override a handler for SIGINT; however that is no the case (Python 2.7.11 with Twisted 16.1.1), the process quits with "KeyboardInterrupt" on stdout.
other resources suggest reactor.run(installSignalHandlers=False) or reactor.run(installSignalHandlers=0), but this appears to have no effect.

Thus I have several questions:

what is the ideologically correct way to handle signals? (if it is possible at all)
what is the recommended way to implement such "state introspection" facilities for Twisted servers? (I was thinking of making it listen for TCP connections on another port, and use that as an alternative for a POSIX signal - but I feel that I am making things too complicated).

Thank you for taking your time to read this, I am looking forward to hints from the hive mind of this planet.

This is the relevant excerpt from the mailing list:

> If you do this, you'll break spawnProcess.  Fortunately, if you just
> install a SIGINT handler, Twisted won't stomp on it:
>    exarkun at charm:~$ python
>    Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)    [GCC 4.2.3 
> (Ubuntu 4.2.3-2ubuntu7)] on linux2
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> def f(*a):
>    ...     print 'sigint'
>    ...    >>> import signal
>    >>> signal.signal(signal.SIGINT, f)
>    
>    >>> from twisted.internet import reactor
>    >>> reactor.run()
>    sigint
>    sigint
>    sigint
>    Quit

Jean-Paul Calderone Jean-Paul Calderone · Accepted Answer · 2017-03-20T12:29:45

As of Twisted 17.1, only twistd installs a SIGUSR1 handler and it will skip the installation if you have already installed a handler for that signal.

The message you see on stdout when you send SIGUSR1 to your process:

user signal 1 received

Is the OS default system handler. The behavior of that default is to exit the process. This suggests that neither you nor Twisted has installed a handler for SIGUSR1. It seems likely that your handler installation code just didn't get run when you thought it was being run and that you're not using twistd.

However...

what is the ideologically correct way to handle signals? (if it is possible at all)

It's quite possible that it's not possible at all. Signals are a rabbit hole. If you have any choice in the matter, stay away from them. Introducing asynchronous signal delivery into any Python program is fraught. It may be possible to do correctly and reliably in some cases but I wouldn't bet on it. Signals have complicated interactions with multithreading because of POSIX shortcomings and additional complications Python adds on top of these. Signal handlers are a shared global resource (as you've noted) so there are complicated interactions between different codebases trying to use them.

Finally, there are a wide array of alternatives available to you which side-step these issues.

what is the recommended way to implement such "state introspection" facilities for Twisted servers? (I was thinking of making it listen for TCP connections on another port, and use that as an alternative for a POSIX signal - but I feel that I am making things too complicated).

Using a mechanism other than POSIX signals may seem more complicated but I would propose that it is strictly simpler, though it may require slightly more typing.

I don't have a general proposal for "state introspection" as such but here is a specific proposal for this case: add an HTTP server to your process.

As example of how straightforward this is:

from json import dumps

from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet.endpoints import serverFromString

class StateResource(Resource):
    def render(self, request):
        return dumps({"your": "state info"})

root = Resource()
root.putChild(b"", StateResource())
serverFromString("tcp:12345").listen(Site(root))

Yep, it's like 10 lines longer than the equivalent signal-based code. However, it has no weird multithreading or multiprocess interactions. It's platform independent. It doesn't stomp on shared resources (nor will it be stomped on) - except for that tcp port I randomly picked (and if you don't like that, you can probably run it over a UNIX socket instead - also granting access control, if that matters to you).

Where this really starts to become obviously the superior solution is when you want to add a health check to your application so your supervisor can monitor it. And then you want to add a metrics reporter so your monitoring system can pull in that data and present it to your SREs or ops team. And so on. A signal-based solution which can deal with that additional functionality becomes even more complex and, I suspect, ends up with a total SLOC count that's higher than the HTTP solution. ie, an HTTP server scales well with new feature requirements. A signal-based solution does not.

How to add a SIGUSR1 handler to a Twisted program?

1 Answers