FIFO with one READER and multiple WRITER in BASH/SH

Question

I have one named pipe (FIFO) say : 'MY_FIFO'

One process reads the fifo from multiple processes to get instructions.

(1) reader bash script:

while read LINE < 'MY_FIFO'
do
 # process line
done

(many) writer bash script:

while [ 1 ]
do
  INSTRUCTION = # get an instruction
  echo $INSTRUCTION > 'MY_FIFO'
done

This works fine to pass instructions to the reader when we have 1 writer or that the writers don't write at the same time. But if we have many writers writing at the same time, the reader gets the first message from whoever was the very first to write to the FIFO, and then hangs waiting for input instead of fetching the next instruction.

What happens behind the scene that prevents me from having multiple writers write to the same fifo? Are they overwriting each other's instructions?

Is there a way for me to test that the reader is ready to read from fifo before writing to the fifo?

Would be able to create a real minimal reproducible example? — KamilCuk
Not really a bash or sh question specifically; this is a problem with FIFO semantics, you'd have it writing the same code in C too. — Charles Duffy
(note, for purposes of determining what constitutes "the same code", that when you exit the code that a redirection applies to, the file handle is explicitly closed, and an entirely new file handle is opened next time you enter that code block). — Charles Duffy
...anyhow, one easy way to address this without changing much would be to have a flock-style exclusive lock that needs to be held before any writer opens the file. — Charles Duffy
@willer2k, ...btw, as an aside, note that all-caps variable names are designated by the POSIX specification for variables that modify shell or system utility behavior, whereas variable names with at least one lower-case character are reserved "for application use", meaning they're guaranteed not to overwrite something that modifies shell behavior. See pubs.opengroup.org/onlinepubs/9699919799/basedefs/…, keeping in mind that environment variables and regular shell variables share a namespace (setting a shell variable will overwrite any like-named environment variable). — Charles Duffy

Ted Lyngmo Ted Lyngmo · Accepted Answer · 2020-10-22T16:21:14

One problem is that when the writer has written its message, it closes the pipe and then your reader will exit that loop. You will also get garbled messages in the pipe when more than one process writes at the same time.

You could try something this instead. It uses flock to lock the pipe while writing to it:

reader

#!/bin/bash

while [[ 1 ]]
do
    IFS= read -r LINE < MY_FIFO
    echo reader got "$LINE"
done

writers - Note that this opens the pipe first, then waits for the lock to succeed and finally writes the message:

#!/bin/bash

# getting instructions from stdin for demo purposes
while IFS= read -r INSTRUCTION
do
    echo Sending "$INSTRUCTION"

    # advisory lock of the pipe when writing
    flock MY_FIFO echo "$INSTRUCTION" > MY_FIFO
done

Optional writers - This creates a lockfile and does not open the fifo until the lock is actually held.

#!/bin/bash

while IFS= read -r INSTRUCTION
do
    echo Sending "$INSTRUCTION"
    {
         flock 9
         echo "$INSTRUCTION" > MY_FIFO
    } 9> MY_FIFO.lck
done

FIFO with one READER and multiple WRITER in BASH/SH

1 Answers