ZeroMQ share a context with all child processes in a forking server

3
votes

I'm writing a forking chat server in C++, with each incoming client being its own process. The server-client interactions are done though normal sockets with ZeroMQ sockets handling message queuing and IPC. The basic problem is that there when the server forks to accommodate the new client, the client's process has a copy of the context (which is what fork does, right?), so when it binds sockets with the context, none of the other clients are aware of the socket. Long story short: how do I get each client thread to have the same context so that they can talk to each other through ZeroMQ?

I've looked at various ways to share the context between processes, and so far I'm found only this one. The problem with that is 1) it uses a thread pool, and from what I understand from what's written there, only 5 threads are created; this server is needs to support at least 256 and thus will have at least that many threads, and 2) it uses ZeroMQ for both talking to clients and for backend tasks; I'm limited to using ZeroMQ for backend only.

I've looked at the ZeroMQ mailing list and one message said that fork() is orthogonal to how ZeroMQ works. Does this mean that I can't share my context across forked child processes? If that's the case, how do I share the context across multiple processes while keeping in mind the requirement of supporting at least 256 clients and using ZeroMQ for only backend?

EDIT: Cleared up the thread/process confusion. Sorry about that.

EDIT2: The reason why I'm also favoring forking over threads is that I'm used to having a main process that accepts an incoming socket connection then forks, giving the new socket to the child. I'm not sure how to do that in a threading fashion (not really well-practiced, but not totally out of my league)

EDIT3: So, starting to rewrite this with threads. I guess this is the only way?

EDIT4: For further clarification, incoming connections to the server can be either TCP or UDP and I have to handle which type it is when the client connects, so I can't use a ZeroMQ socket to listen in.

1
c++forkipczeromq
In your question you are confusing threads with processes quite a lot. They are not the same thing at all, please correct to make it clearer. – syam
If you have to use processes, there are few IPC methods available: sockets or shared memory come to my mind. But since you need to share context, a single process with multiple threads would be much better than separate processes though. – syam
Re. Edit 3: Threads are definitely not the only way, processes could work too with some decent IPC, it's just a matter of the effort you want to invest. On the other hand multi-threading can be quite confusing when you are not used to it, but it is very rewarding when you start to understand it so I believe it is well worth the effort. :) I for one am used to multithreading and I wouldn't exchange this paradigm for anything in the world... – syam
To settle the matter: we all know there isn't ONE true way. The reason why I direct you toward multithreading is because you obviously need to share context, and this is done more easily with multithreading and some synchronization than with IPC. Of course this means a paradigm shift for you from the traditional fork model but it is rewarding enough to easily justify the learning you'll have to face. Good luck, and if you need further advice (whatever it is, even if it's "just" how to switch from fork to threads) rest assured that SO will still be here for you. ;) – syam
Thanks! Nose to the grindstone at the moment trying to get the threading architecture up, but once that's up things should go a lot smoothly. – Dreamless Memory

1 Answers

6
votes

Context Sharing

The reason to share ZMQ context in the example code from your link is, that the server(main()) uses inproc socket to communicate with workers(worker_routine()). Inproc sockets cannot communicate with each other unless they are created from the same ZMQ context, even they settle in the same process. In your case, I think it's not necessary to share it since no inproc sockets are supposed to be used. So, your code might look like:

void *worker_routine (void *arg)
{
    // zmq::context_t *context = (zmq::context_t *) arg;    // it's not necessary for now.
    zmq::context_t context(1);    // it's just fine to create a new context

    zmq::socket_t socket (context, ZMQ_REP);
    // socket.connect ("inproc://workers");    // inproc socket is useless here.
    socket.connect("ipc:///tmp/workers");    // need some sockets who can cross process.

    // handling code omitted.
}

int main ()
{
    //  omitted...

    // workers.bind ("inproc://workers");    // inproc socket is useless here.
    workers.bind("ipc:///tmp/workers");

    //  Launch pool of worker processes
    for (int i = 0; i < 5; ++i) {
        if (fork() == 0) {
            // worker process runs here
            worker_routine(NULL);
            return 0;
        }
    }
    //  Connect work processes to client process via a queue
    zmq::proxy (clients, workers, NULL);
    return 0;
}

handling process per request

And now to talk about your requirement, one process per request. The last example code is just intended to illustrate the usage of zmq::proxy, which is provided to simplify the server code with ROUTER-DEALER pattern. But it can't fulfill your requirement. So, you have to implement it manually. It just looks like another example. The difference is that you need to invoke fork() when frontend socket is readable and put the while loop into sub process.

if (items[0].revents & ZMQ_POLLIN) {
    if (fork() == 0) {
        // sub process runs here
        while (1) {
            // forward frames here
        }
        // sub process ends here
        return 0;
    }
}

Suggestion

At the end, I have to say, it's too much heavy to create a process for one request exactly unless your scenario is really special. Please use thread, or consider the asynchronous IO like zmq::poll.