Python <p = Popen(["command"], stdout=PIPE)> when p.stdout.read(-1) is blocked

Question

hello i have question in Python subprocess module

I want to interaction with child process (ex: child's stdin, stdout)

This is child Program's code

  1 #include <stdio.h>
  2 
  3 int main(){
  4         char buf[1024];
  5         printf("asdfsadfasff\n");
  6         scanf("%s",buf);
  7         printf("%s\n",buf);
  8 }

this code is very simple, print the string, wait input, and print input string

and next, the python code is below

  1 #!/usr/bin/python
  2 from subprocess import *
  3 
  4 fd = Popen(["./random"],stdin=PIPE,stdout=PIPE)
  5 print "pid = %d"%(fd.pid) # print pid
  6 result = ""
  7 result = fd.stdout.read(-1) # read
  8 print result
  9 fd.stdin.write("write write write!!!\n")
 10 result = fd.stdout.read(-1)
 11 print result

in this case, I expected this program will work well, but in line 7 (fd.stdout.read(-1) was blocked and never working

I changed read function's param (read(-1), read(1), read(), read(1024)) but everything do not work

But, when i give string in stdin first, program worked.

I think child's stdout buffer was not fill when the program is ended... this is just my opnion

Is there any solution about this problem?

Edit1. when I execute program which is print string first, and wait user's input such as "sudo su" This worked well that I expected

I cannot understand why this case work well

  1 from subprocess import *
  2 
  3 fd = Popen(["sudo","su"],stdin=PIPE,stdout=PIPE)
  4 
  5 print fd.stdout.read(-1)
  6 print fd.stdin.write("asdfasdfas\n")

what happens if you flush(stdout); after your first printf? — mgilson
Apparently the C function is fflush...sorry, my C is a little rusty. — mgilson
1 #include <stdio.h> 2 3 int main(){ 4 char buf[10]; 5 printf("asdfsadfasff\n"); 6 fflush(stdout); 7 scanf("%s",buf); 8 printf("%s\n",buf); 9 } add fflush, but result is not changed, is there something mistake in timeing of call fflush? — Joonsung Kim
@JoonSunKim without the line numbers, it's still valid C code ;). It looks like what I had envisioned though. You're also passing a string that is too long for your character buffer ( stackoverflow.com/a/2430310/748858 ). That could result in some funny behavior as well. — mgilson

torek torek · Accepted Answer · 2012-05-15T19:27:13

The child process ./random is using stdio, which by default does "full buffering" rather than "line buffering" when the process's stdout is not a tty device. Since python runs the process via pipes (which are not tty devices) the program is buffering its output, and not printing it at the time of the scanf call. The buffer is flushed (written to stdout) only (a) when it fills up; (b) when explicitly flushed with an fflush call; or (c) at program exit.

You can modify the child process to do explicit fflush calls, or you can use a pseudo-tty from Python, e.g., with pexpect or sh. (Note that the sh module by default uses a pty for output only but this is usually sufficient.) Or, in cases like this where the program you're running only "looks interactive" (but is actually totally predictable), you can just run it with the known input sequence pre-provided:

fd = Popen(["./random"], stdin=PIPE, stdout=PIPE)
result = fd.communicate("write write write!!!\n")[0]

although then of course you'll get all the stdout input mixed together, and have to separate out the two lines it printed (in this particular case).

Python <p = Popen(["command"], stdout=PIPE)> when p.stdout.read(-1) is blocked

1 Answers