python subprocess pipe unbuffered behaviour

Question

I've the below piece of code to read data from a child process as its generated and write to a file.

from subprocess import Popen, PIPE
proc = Popen('..some_shell_command..', shell=True, stdout=PIPE)
fd = open("/tmp/procout", "wb")
while True:
    data = proc.stdout.read(1024)
    if len(data) == 0:
        break
    fd.write(data)
fd.close()

'Popen' default bufsize is 0 => unbuffered. What will happen if for some reason the write-to-file operation experiences a huge latency?

Assuming that the child process is supposed to produce 500GB of data, do all those data get stored in memory until the parent reads them all? (OR)
Will the child process wait for 1024 bytes of data to be read by the parent before writing the next 1024 bytes to stdout? (OR)
Will the child process wait after the OS pipe buffer gets filled and once the parent reads, the child resumes writing again? (OR)
??

frist frist · Accepted Answer · 2016-07-18T09:35:05

Answering to your questions:

No, it will not be stored in memory. The child process will stuck on write operation after exceeding pipe-max-size limit (cat /proc/sys/fs/pipe-max-size);
The child process will write about 1M before it will stuck, until the parent process read block of data. After this child process will write next 1024 bytes sequentualy as fast as they will be readed;
Yes in case of blocking IO the process will be blocked by the OS when write syscall will be called. In case of non-blocking IO I hope write syscall will return EAGAIN or other system-specific error.

So actually the application will stuck while calling write system call waiting for the pipe buffer will available. It doesn't mean that it will hang. For example if an application implements some kind of internal queue and it have more than one thread, it can continue to work and add any data to it's queue while the writting-out thread will wait for the buffer.

python subprocess pipe unbuffered behaviour

1 Answers