Python subprocess blocks in embedded environment

Question

My script takes large number of files as input, calling external programs using subprocess.call(). It works perfectly from regular OS consoles, but hangs after processing 10 files or so when running in an embedded environment using its embedded Python 2.7.x.

I've referred to various similar questions and failed to find one working for me:

Python: subprocess.Popen and subprocess.call hang

Python subprocess hangs

Python subprocess call hangs

and this in-depth discussion: https://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/

They all imply the vulnerable buffered PIPE, and suggest using file-like objects for stdout and stderr. So I also added a temporary text file, opened it, and fed it to stdout and stderr of subprocess.call(). It didn't work either.

My old code is fairly straight forward:

# script1.py

folder = "/path/to/my/folder"
for root, dirs, files in os.walk(folder):
    for file in files:
        path = join(root, file)
        try:
            cmd = ['dir', path, '1>&2']
            _logger.debug(' '.join(cmd))
            completed = subprocess.call(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
        except subprocess.CalledProcessError as err:
            _logger.debug(err)
        else:
            _logger.debug('returncode: {}'.format(completed))
print('all done!!')

the main script:

try:
    cmd = ['python', 'script1.py', '1>&2']
    ue.log(' '.join(cmd))
    completed = subprocess.call(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as err:
    _logger.error(err)

I've mixed-and-matched a couple of stdout/stderr solutions. including adding:

with open(join(_script_dir, 'tmp.txt'), 'w') as tmp:
    #
    # old code
    #
     completed = subprocess.call(cmd, shell=True, stdout=tmp, stderr=tmp)

None worked.

Answer 1

I ended up throwing away all the pipes and stdout/stderr, and use the default of Python 3.7.2's subprocess.Popen() arguments, i.e., None for stdout/stderr, and remove 1>&2. So basically just don't bother retrieving the realtime output of another program when the program works with large amount of input and potentially generates a huge wave of output through the narrow and hidden pipe. I turned to using just log files written for each individual script.

I am much happier with this solution than blindly trying all kinds of UNIX tricks to solve the pipe issue on Windows.

P.S., Some answers from my above links suggested shell=False, I found that annoying in that the embedded environment will spawn lots of floating console windows out of each subprocess.call().

Answer 2

There is a solution to the hanging pipe problem which is to use from multiprocessing import Manager You set up a task queue:

mgr = Manager()

task_queue = mgr.Queue()

Pass it to the process as an argument:

gmat_args.append([gmat_arg, task_queue])
...
pool = Pool(processes=nrunp, maxtasksperchild=20)
...
results = pool.map(run_gmat, gmat_args, chunksize=ninstances)

The process writes to the pool:

def run_gmat(args):
    q = args[1]

    scriptname = os.path.basename(args[0])

    proc = sp.Popen(['gmat', '-m', '-ns', '-x', '-r', str(args[0])])

    (outs, errors) = proc.communicate(timeout=cpto)

    outs = outs.decode('UTF-8')

    q.put(filter_outs(outs, scriptname))

Then back in the main proc, you read the queue and log it:

while 1:

    qout = task_queue.get(cpto)   

    logging.info(qout)

    if task_queue.qsize() < 1:

        break

I can't say this is perfect, I'm running over 4000 jobs and I get 2 - 3 timeouts which seem to be related to file I/O (the job writes a massive report file on completion). To prevent hanging I catch a timeout exception, log the final stdout, stderr to the queue and kill the job. I lose the report files but I can see the timeout in the log from the queue, so it's simply a matter of rerunning those 2-3 jobs.