How can I use pthreads to read from many running processes at once?

Question

I'm using pthreads in a "work crew" pattern where I have a large number of commands to run, and I want to read the output from each command and merge all the outputs. Using pthreads I create one worker thread per core, and each worker thread essentially does the following:

for (;;) {
   const char *command = get_from_work_queue();
   if (command == NULL)
       break;
   FILE *fp = popen("long-running-command", "r");
   // ... loops reading fp until EOF
   // enqueue data read from fp
}

Both get_from_work_queue and "enqueue data" require the thread to temporarily acquire a mutex, as the queues are shared, but these mutexes are released before any call to popen() or fread().

My issue is that although I have four cores and four worker threads, only one long-running command is going at any one time. The other three threads are just sitting around, probably waiting to be scheduled.

I assume that something about popen or fread is blocking all the threads, and not just the caller. How can I read from a command without blocking other threads, so I can have four long-running commands going at the same time?

The accepted answer here: popen() alternative may be worth a try to see if you get better/desired behavior. — Phil Brubaker
@PhilBrubaker I can't quite see why forking an extra shell would make a difference---and the truth is I need to run the shell. But it could be worth a try. — Norman Ramsey
@EdHeal coding with select(2) is a huge pain in the ass. I was really hoping for threads with non-blocking I/O. But I may be SOL. — Norman Ramsey
A simple test shows that having multiple popen() commands running simultaneously from multiple threads works fine. If you backtrace your threads in gdb, where are they stopped? — caf

Norman Ramsey Norman Ramsey · Accepted Answer · 2017-04-24T01:42:33

Turns out it works fine if I use read(2) instead of fread(3). For slightly more detail see my answer to What multithreading package for Lua "just works" as shipped?

How can I use pthreads to read from many running processes at once?

1 Answers