Live stdout output from Python subprocess in Jupyter notebook

30
votes

I'm using subprocess to run a command line program from a Python (3.5.2) script, which I am running in a Jupyter notebook. The subprocess takes a long time to run and so I would like its stdout to be printed live to the screen in the Jupyter notebook.

I can do this no problem in a normal Python script run from the terminal. I do this using:

def run_command(cmd):
from subprocess import Popen, PIPE
import shlex

with Popen(shlex.split(cmd), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
    for line in p.stdout:
        print(line, end='')
    exit_code = p.poll()
return exit_code

However, when I run the script in a Jupyter notebook, it does not print the stdout live to the screen. Instead, it prints everything after the subprocess has finished running.

Does anyone have any ideas on how to remedy this?

Many thanks, Johnny

4
pythonsubprocessipythonstdoutjupyter
Try adding flush=True to your print - Padraic Cunningham
Thanks, Padraic. I tried that, but it didn't work. I also tried adding sys.stdout.flush() before the for loop, and that didn't work either. - Johnny Hunter
In what system and version of Jupyter are you running this? I ran your code with jupyter_client 4.3.0, jupyter_console 5.0.0 and jupyter_core 4.1.0 under Ubuntu and Python 3, and stdout was progressively printed as the process was generating it. - foglerit
What command are you running? - Padraic Cunningham
@jonnat, thanks! My versions are: jupyter==1.0.0 jupyter-client==4.3.0 jupyter-console==5.0.0 jupyter-core==4.1.0 - Johnny Hunter

4 Answers

29
votes

The ipython notebook has it's own support for running shell commands. If you don't need to capture with subprocess stuff you can just do

cmd = 'ls -l'
!{cmd}

Output from commands executed with ! is automatically piped through the notebook.

4
votes

If you set stdout = None (this is the default, so you can omit the stdout argument altogether), then your process should write its output to the terminal running your IPython notebook server.

This happens because the default behavior is for subprocess to inherit from the parent file handlers (see docs).

Your code would look like this:

from subprocess import Popen, PIPE
import shlex

def run_command(cmd):
    p = Popen(shlex.split(cmd), bufsize=1, universal_newlines=True)
    return p.poll()

This won't print to the notebook in browser, but at least you will be able to see the output from your subprocess asynchronously while other code is running.

Hope this helps.

4
votes

Jupyter mucks with stdout and stderr. This should get what you want, and give you a more useful exception when the command fails to boot.

import signal
import subprocess as sp


class VerboseCalledProcessError(sp.CalledProcessError):
    def __str__(self):
        if self.returncode and self.returncode < 0:
            try:
                msg = "Command '%s' died with %r." % (
                    self.cmd, signal.Signals(-self.returncode))
            except ValueError:
                msg = "Command '%s' died with unknown signal %d." % (
                    self.cmd, -self.returncode)
        else:
            msg = "Command '%s' returned non-zero exit status %d." % (
                self.cmd, self.returncode)

        return f'{msg}\n' \
               f'Stdout:\n' \
               f'{self.output}\n' \
               f'Stderr:\n' \
               f'{self.stderr}'


def bash(cmd, print_stdout=True, print_stderr=True):
    proc = sp.Popen(cmd, stderr=sp.PIPE, stdout=sp.PIPE, shell=True, universal_newlines=True,
                    executable='/bin/bash')

    all_stdout = []
    all_stderr = []
    while proc.poll() is None:
        for stdout_line in proc.stdout:
            if stdout_line != '':
                if print_stdout:
                    print(stdout_line, end='')
                all_stdout.append(stdout_line)
        for stderr_line in proc.stderr:
            if stderr_line != '':
                if print_stderr:
                    print(stderr_line, end='', file=sys.stderr)
                all_stderr.append(stderr_line)

    stdout_text = ''.join(all_stdout)
    stderr_text = ''.join(all_stderr)
    if proc.wait() != 0:
        raise VerboseCalledProcessError(proc.returncode, cmd, stdout_text, stderr_text)
1
votes

Replacing the for loop with the explicit readline() call worked for me.

from subprocess import Popen, PIPE
import shlex

def run_command(cmd):
    with Popen(shlex.split(cmd), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
        while True:
            line = p.stdout.readline()
            if not line:
                break
            print(line)    
        exit_code = p.poll()
    return exit_code

Something is still broken about their iterators, even 4 years later.