How to redirect output with subprocess in Python?

103
votes

What I do in the command line:

cat file1 file2 file3 > myfile

What I want to do with python:

import subprocess, shlex
my_cmd = 'cat file1 file2 file3 > myfile'
args = shlex.split(my_cmd)
subprocess.call(args) # spits the output in the window i call my python program
5
pythonsubprocess
Executing such a command in subprocess would not give you any output. May be you want to run it without > myfile redirecting output from cat file1 file2 file3 into python? - PoltoS
@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it? - catatemypythoncode
os.sendfile()-based solution is possible, see Reproduce the unix cat command in python - jfs
I think that output redirection ('>' or '>>') doesn't work in subprocess.Popen (at least in Python 2.7) (in shell=True mode) In this example, as others point out, you can work around this by not using redirection, but in other cases redirection is useful. If redirection or piping is not supported in subprocess.Popen is should be documented (and/or os.system() should be not be deprecated until this is fixed) - Ribo

5 Answers

19
votes

UPDATE: os.system is discouraged, albeit still available in Python 3.

Use os.system:

os.system(my_cmd)

If you really want to use subprocess, here's the solution (mostly lifted from the documentation for subprocess):

p = subprocess.Popen(my_cmd, shell=True)
os.waitpid(p.pid, 0)

OTOH, you can avoid system calls entirely:

import shutil

with open('myfile', 'w') as outfile:
    for infile in ('file1', 'file2', 'file3'):
        shutil.copyfileobj(open(infile), outfile)
306
votes

In Python 3.5+ to redirect the output, just pass an open file handle for the stdout argument to subprocess.run:

# Use a list of args instead of a string
input_files = ['file1', 'file2', 'file3']
my_cmd = ['cat'] + input_files
with open('myfile', "w") as outfile:
    subprocess.run(my_cmd, stdout=outfile)

As others have pointed out, the use of an external command like cat for this purpose is completely extraneous.

5
votes

@PoltoS I want to join some files and then process the resulting file. I thought using cat was the easiest alternative. Is there a better/pythonic way to do it?

Of course: 

with open('myfile', 'w') as outfile:
    for infilename in ['file1', 'file2', 'file3']:
        with open(infilename) as infile:
            outfile.write(infile.read())
1
votes
size = 'ffprobe -v error -show_entries format=size -of default=noprint_wrappers=1:nokey=1 dump.mp4 > file'
proc = subprocess.Popen(shlex.split(size), shell=True)
time.sleep(1)
proc.terminate() #proc.kill() modify it by a suggestion
size = ""
with open('file', 'r') as infile:
    for line in infile.readlines():
        size += line.strip()

print(size)
os.remove('file')

When you use subprocess , the process must be killed.This is an example.If you don't kill the process , file will be empty and you can read nothing.It can run on Windows.I can`t make sure that it can run on Unix.

0
votes

One interesting case would be to update a file by appending similar file to it. Then one would not have to create a new file in the process. It is particularly useful in the case where a large file need to be appended. Here is one possibility using teminal command line directly from python.

import subprocess32 as sub

with open("A.csv","a") as f:
    f.flush()
    sub.Popen(["cat","temp.csv"],stdout=f)