bash: force exec'd process to have unbuffered stdout

25
votes

I've got a script like:

#!/bin/bash
exec /usr/bin/some_binary > /tmp/my.log 2>&1

Problem is that some_binary sends all of its logging to stdout, and buffering makes it so that I only see output in chunks of a few lines. This is annoying when something gets stuck and I need to see what the last line says.

Is there any way to make stdout unbuffered before I do the exec that will affect some_binary so it has more useful logging?

(The wrapper script is only setting a few environment variables before the exec, so a solution in perl or python would also be feasible.)

6
bashloggingexecunbuffered-output
Dennis's answer is exactly right. Here's why. Programs on Unix that use C stdio for output will buffer when the output is not going to a terminal (boosts performance a lot) unless told not to with setvbuf; hardly anyone bothers with that. The expect system (of which unbuffer is a trivial application) uses black magic with Unix pseudo-terminal devices (ptys) to run programs with a terminal that isn't a real terminal so that the wrapped program doesn't buffer its output. It's clever, but be warned: most systems only have a limited number of ptys; a naturally unbuffered program is better. – Donal Fellows
BTW, this doesn't work so well on Windows. That's because expect on that platform uses debugging magic instead of terminal magic (because this is an area where Windows is very different). – Donal Fellows
Thanks. I decided to opt for modifying some_binary to do the right thing instead of relying on the black magic. expect is great, but I don't want it here... – bstpierre

6 Answers

34
votes

GNU coreutils-8.5 also has the stdbuf command to modify I/O stream buffering:

http://www.pixelbeat.org/programming/stdio_buffering/

So, in your example case, simply invoke:

stdbuf -oL /usr/bin/some_binary > /tmp/my.log 2>&1

This will allow text to appear immediately line-by-line (once a line is completed with the end-of-line "\n" character in C). If you really want immediate output, use -o0 instead.

This way could be more desirable if you do not want to introduce dependency to expect via unbuffer command. The unbuffer way, on the other hand, is needed if you have to fool some_binary into thinking that it is facing a real tty standard output.

14
votes

You might find that the unbuffer script that comes with expect may help.

11
votes

Some command line programs have an option to modify their stdout stream buffering behaviour. So that's the way to go if the C source is available ...

# two command options ...
man file | less -p '--no-buffer'
man grep | less -p '--line-buffered'

# ... and their respective source code

# from: http://www.opensource.apple.com/source/file/file-6.2.1/file/src/file.c
if(nobuffer)
   (void) fflush(stdout);

# from: http://www.opensource.apple.com/source/grep/grep-28/grep/src/grep.c
if (line_buffered)
   fflush (stdout);

As an alternative to using expect's unbuffer script or modifying the program's source code, you may also try to use script(1) to avoid stdout hiccups caused by a pipe:

See: Trick an application into thinking its stdin is interactive, not a pipe

# Linux
script -c "[executable string]" /dev/null

# FreeBSD, Mac OS X
script -q /dev/null "[executable string]"
3
votes

I scoured the internets for an answer, and none of this worked for uniq which is too stubborn to buffer everything except for stdbuf:

{piped_command_here} | stdbuf -oL uniq | {more_piped_command_here}

2
votes

GNU Coreutils-8 includes a program called stdbuf which essentially does the LD_PRELOAD trick. It works on Linux and reportedly works on BSD systems.

2
votes

An environment variable can set the terminal IO mode to unbuffered.

export NSUnbufferedIO=YES

This will set the terminal unbuffered for both C and Ojective-C terminal output commands.