Python subprocess call hangs

Question

Python version: 2.6.7 I have the following subprocess.call within a for loop which is exectuted 18 times, however, the process constantly hangs on the 19th loop:

if config.get_bool_option(NAME, 'exclude_generated_code', True):
            for conf in desc.iter_configs():
                for gen in desc.iter_generators(conf):
                    generator.initialize_generated_path(gen, desc)
                    for genpath in gen.generated_path:
                        os.rename(cov_file, cov_file+'.temp')
                        exclude = ['lcov']
                        exclude += ['-r', cov_file+'.temp', '"'+genpath+'/*"']
                        exclude += ['-o', cov_file]
                        if verbose: Tracer.log.info("Running "+ ' '.join(exclude))
                        try:
                            subprocess.call(' '.join(exclude), stdout=out, stderr=out, shell=True)
                        except subprocess.CalledProcessError, e:
                        if verbose: Tracer.log.info("TESTING: Got Exception \n")

The console output looks as follows:

Running lcov -r /remote/XXXXXX/coverage.19.temp "/remote/XXXXXX/xml/2009a/generated/*" -o /remote/XXXXX/gcov/coverage.19

Since I am not very familiar with python scripts, I just wandered whether I am doing something wrong here...I suspect a deadlock somewhere..

Would the stdout, stderr = process.communicate() deal with these issues?

Any expert answer on in which cases would the subprocess.call hang please? Thanks very much

Eric Hydrick Eric Hydrick · Accepted Answer · 2013-08-16T13:50:17

When using subprocess, I tend to do something like this:

is_running = lambda: my_process.poll() is None

my_process = subprocess.Popen(' '.join(exclude), 
                              stdout=subprocess.PIPE, 
                              stderr=subprocess.PIPE,
                              shell=True)

# Grab all the output from stdout and stderr and log it
while is_running():
    rlist, wlist, xlist = select.select([my_process.stdout, my_process.stderr], [], [], 1)

# Log stdout, but don't spam the log
if my_process.stdout in rlist and verbose:
    # Adjust the number of bytes read however you like, 1024 seems to work 
    # pretty well for me. 
    Tracer.log.debug(my_process.stdout.read(1024))

# Log stderr, always
if my_process.stderr in rlist:
    # Same as with stdout, adjust the bytes read as needed.
    Tracer.log.error(my_process.stderr.read(1024))

I've seen the stdout stuff just dump a bunch of empty lines in my logs in the past, which is why I log that at the debug level. That prints to my logs during the development, but never gets written in production, so I can safely leave it in the code for debugging without putting garbage in their logs.

Hopefully, this can help expose just where your program is hanging and what's causing it.

Python subprocess call hangs

1 Answers