Multiple reactors (main loops) in one application through threading (or alternative means)

Question

I've got an idea for an app I'd like to work on to learn a bit more about Twisted and WebSockets. I was thinking of integrating a previously written IRC Bot into a web application. As far as I can see it, I would need three reactors to make it work:

Primary Reactor: Web Server (HTTP). This would be your average twisted.web application. When you access it, you can POST an IRC server/channel to connection. The web server would then talk to a different reactor in a different thread, which is...
Secondary Reactor: IRC Bot. This would be an IRC bot running via the Twisted IRC client protocol. It would join a channel, and whenever something was said, it would take that data and push it to yet another reactor, on yet another thread, which is...
Tertiary Reactor: WebSocket Server (WS): Since WebSockets don't use the regular HTTP Protocol, they need their own server (or so it seems, looking at examples such as this. When the IRC bot receives a message, it tells the WebSocket Server to push that message to connected clients.

In my mind, this makes sense. It seems like it would be possible. Does anyone have any examples of multiple reactors running in separate threads, or is this something I've imagined that can't be done in the current incarnation of Twisted.

Are there any architecture changes that can (or should) be made to minimize the reactor count, etc?

Thanks for any help.

You have the tag multithreading assigned. Keep in mind that no multithreading will occur magically in the same process from using the same reactor with multiple factories on different ports (and you're supposed to use only one reactor per process). If your Web Server blocks, it will block the IRC Bot and the WebSocket Server as well. There's not only the GIL, but also the event-driven nature of Twisted. — Daniel F

Glyph Glyph · Accepted Answer · 2010-11-03T10:25:57

Lucky for you, it is easy to reduce the number of reactors, specifically, to 1:

You can only ever have a single reactor, in a single thread, in any given Twisted process. If you try to have more, nothing will work.

The whole point of a reactor, actually, is to facilitate having multiple sources of events combined into a single thread. If you want to listen on 3 different ports with 3 different protocols, your application might look like this:

from twisted.internet import reactor
reactor.listenTCP(4321, FirstProtocolFactory())
reactor.listenTCP(5432, SecondProtocolFactory())
reactor.listenTCP(6543, ThirdProtocolFactory())
reactor.run()

Of course, you may not actually be calling listenTCP directly yourself, as you probably want to use Service objects from twisted.application.internet if you are using twistd, either via a .tac file or a twistd plugin. And you won't need to call reactor.run() yourself, if twistd is doing it for you. My point here is that, via whatever means, you load up the reactor with all the events you expect it to react to - listening servers, client connections, timed events - and it will react to each one as it occurs. (Hence, "reactor".)

For the specific values of what FirstProtocolFactory, SecondProtocolFactory, and ThirdProtocolFactory should be, see the links in pyfunc's answer.

Multiple reactors (main loops) in one application through threading (or alternative means)

2 Answers