Why stdbuf has no effect on Python?

Question

Given the following program in Python:

import sys

print("input")
while True:
    pass

and the equivalent in C:

#include <stdio.h>

int main() {
        printf("input\n");
        while(1);
        return 0;
}

When I pipe the program with cat, the output is by default buffered (instead of line-buffered). As a consequence, I have no output:

% python3 ./a.py | cat
(empty)

% ./a.out | cat
(empty)

I can use stdbuf to switch back to line-buffering:

% stdbuf -oL ./a.out | cat
input

But that doesn't work with Python:

% stdbuf -oL python3 a.py | cat
(empty)

Any idea why? I'm aware of the existence of python3 -u but I want line-buffering, not "no buffering" and I want this command line to work for other languages as well. The command unbuffer seems to work too, but I'd like to understand why stdbuf doesn't work here.

From man stdbuf NOTE: If COMMAND adjusts the buffering of its standard streams ('tee' does for e.g.) then that will override corresponding settings changed by 'stdbuf'. Also some filters (like 'dd' and 'cat' etc.) don't use streams for I/O, and are thus unaffected by 'stdbuf' settings. I guess python is doing this. — Eugene Sh.

John Bollinger John Bollinger · Accepted Answer · 2019-04-12T15:42:51

By default, Python's print() function directs its output to sys.stdout, whose documentation specifies this:

When interactive, stdout and stderr streams are line-buffered. Otherwise, they are block-buffered like regular text files. You can override this value with the -u command-line option.

Note well that those docs do not leave room for general environmental influences, such as that of the stdbuf command, on the buffering mode of sys.stdout: it is unbuffered if the -u option was used (or, equivalently, if environment variable PYTHONUNBUFFERED was set), otherwise line-buffered if interactive and block-buffered if non-interactive.

Programs can control their own buffering, as the docs of stdbuf acknowledge:

NOTE: If COMMAND adjusts the buffering of its standard streams ('tee' does for e.g.) then that will override corresponding settings changed by 'stdbuf'. Also some filters (like 'dd' and 'cat' etc.) don't use streams for I/O, and are thus unaffected by 'stdbuf' settings.

Since Python explicitly specifies buffering details, it is reasonable to expect that it in fact does affirmatively manage its buffering, thus mooting any effect of stdbuf.

Why stdbuf has no effect on Python?

2 Answers