Detect closed non-blocking stdout/stderr pipes in python subprocess

Question

I am using Python to run a process interactively.

p = Popen ("/path/to/my-executable", stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=1)

for f in (p.stdout, p.stderr):
    flags = fcntil.fcntl (f, fcntl.F_GETFL) | os.O_NONBLOCK
    fcntl.ftcntl (f, flags)

for line in sys.stdin:
    p.stdin.write (line.encode ('utf-8'))
    ...

Suppose I read a command from stdin and forward it to my-executable via p, this process may or may not output anything to stdout/stderr, and it may or may not terminate.

I want to wait until my-executable "does something" before reading another command from stdin.

I tried this with bash as a subprocess and "exit" as the line written to its stdin, when I read from its stdout and sterr, stdout gave None and stderr gave b'' but I don't expect this is a reliable indication that a process has finished running (i.e. I cannot reasonably p.wait() because there is nothing in that result which decisively tells me that the process will not give any more output).

How do I correctly make this Python script wait until

something is written to stdout or stderr OR
stdout or stderr is closed

?

Joseph Sible-Reinstate Monica Joseph Sible-Reinstate Monica · Accepted Answer · 2020-06-03T21:00:14

You can use selectors for that, like this:

# one-time setup
import selectors
sel = selectors.DefaultSelector()
sel.register(p.stdout, selectors.EVENT_READ)
sel.register(p.stderr, selectors.EVENT_READ)

# each time you want to wait
sel.select()

I'd recommend that you use unbuffered I/O, though, rather than line-buffered I/O. Also, you can do os.set_blocking(f.fileno(), False) to make your FD's non-blocking, rather than having to mess with fcntl yourself.

Detect closed non-blocking stdout/stderr pipes in python subprocess

1 Answers