How to unblock a thread blocking on a deleted named pipe?

Question

I have a Python program which uses named pipes for various purposes. Each pipe is managed from a different thread so it won't block the main thread.

Let's say I have a thread which is blocking on a call to open('in', 'rb') where in is the relative path of the named pipe. If I wish to shut down my program, I use something like this to unblock my thread from an other one:

with suppress(OSError):
    fd = os.open('in', O_WRONLY | O_NONBLOCK)
    os.close(fd)

This just opens the pipe in write mode, so that the thread blocking on open can move on, then close it. I use O_NONBLOCK to avoid blocking in case the other thread has already terminated (and ignore the potential OSError).

This works fine until someone decides to delete the named pipe in while my thread is blocking on open. In this case I cannot use my "try to open the pipe in non blocking mode and close it" method, because the pipe is no longer visible on the file system (and the non blocking open would just create a brand new pipe).

What is the proper solution to this problem other than killing the thread? Note that I cannot prevent other processes from removing the pipe and permissions won't help (the deleting process could run as root).

Worst case scenario I could use non blocking open to poll the pipe rather than explicitly block on opening it, but I find the blocking approach more elegant, as polling requires wasting CPU cycles and introduces a timing factor. — krispet krispet
Your problem sounds like an XY one (down to why would someone use named pipes in the first place?). But if you set a signal handler, sending the signal will interrupt the blocking open() syscall, allowing your program to continue after handling the EINTR error. So you can send a signal instead of that open/close trick. — mosvy
@mosvy I must use named pipes (long story). Is that possible to do in Python (3.7)? The docs say: "Python signal handlers are always executed in the main Python thread, even if the signal was received in another thread. This means that signals can’t be used as a means of inter-thread communication." — krispet krispet
The signal handler doesn't have to do anything; just to interrupt the open() syscall. See the "answer" (I'm not a python programmer, so that may not be the best approach). Another idea would be to open the named pipe with O_RDWR (which will not block) -- but in that case you cannot determine via EOF when the writer has closed its end of the pipe. — mosvy
It seems that there's no way to a) get the real thread id in python (necessary in order to send a signal to a specific thread) and b) handle EINTR in python3 -- all syscalls will be restarted by hand by the interpreter (that is still possible in python2, though). So, the only way around it is to open the named pipe in read-write mode (which is supported on all modern systems). — mosvy

krispet krispet krispet krispet · Accepted Answer · 2019-08-03T19:26:14

I have solved my problem by using os.open('in', O_RDONLY | O_NONBLOCK), which yields a file descriptor even if there is no writer on the other side of the pipe.

Once I had a valid file descriptor for reading, I was able to feed this to the select() system call to block until there is something to read.

In order to deal with the "what if someone deletes the pipe from the file system while I'm blocking" problem, I've used the pipe() syscall to get an unnamed pipe (the Python version just yields 2 file descriptors for the 2 ends of the pipe).

I feed the read descriptor of this unnamed pipe to the select() call as well, so any time I wish to stop my program I just unblock select() by writing to the write descriptor of the unnamed pipe regardless of what's up with the named pipe.

How to unblock a thread blocking on a deleted named pipe?

1 Answers