Cant use python subprocess to get return value from curl call

Question

So I am trying to write a simple wrapper in python to call rasa, a nlu tool from. The command I would write on the the command line is this:

curl -X POST "localhost:5000/parse" -d '{"q":"I am looking for fucking Mexican food"}' | python -m json.tool

The output I expect is something like this:

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 545 0 500 100 45 33615 3025 --:--:-- --:--:-- --:--:-- 35714

plus the outprint of a json file.

I wrote this program in python:

import subprocess

utterance = "Lets say something"


result = subprocess.run(["curl", "-X", "POST", "localhost:5000/parse", "-d", "'{\"q\":\""+utterance+"\"}'", "|", "python", "-m", "json.tool"], stdout=subprocess.PIPE)
print(vars(result))
print(result.stdout.decode('utf-8'))

Unfortunately my output is like this, meaning I dont actually get the return from the curl call:

{'args': ['curl', '-X', 'POST', 'localhost:5000/parse', '-d', '\'{"q":"Lets say something"}\'', '|', 'python', '-m', 'json.tool'], 'returncode': 2, 'stdout': b'', 'stderr': None}

If I call my python programm from the commandline, this is the output:

curl: option -m: expected a proper numerical parameter curl: try 'curl --help' or 'curl --manual' for more information {'args': ['curl', '-X', 'POST', 'localhost:5000/parse', '-d', '\'{"q":"Lets say something"}\'', '|', 'python', '-m', 'json.tool'], 'returncode': 2, 'stdout': b'', 'stderr': None}

I tried looking everywhere but just cant get it going. Would really appreciate some help.

A thought, have you looked at pycurl or requests -- you would be writing native code, easier to implement, easier to read, easier to debug. — SteveJ

phospodka phospodka · Accepted Answer · 2018-02-27T22:22:11

Update: I grossly misunderstood the question the first time. Rushed reading the details, so my apologies there. You are having a problem because you are trying to pipe two commands together using Popen. The pipe operator, however, is something implemented by the shell, not python. So it is expecting your command to just be a command related to curl. It is complicated, but you have options.

I think for your particular example, the simplest is to not try to chain the command to json.tool. You actually have no need for it. You are already in python, so you can just pretty print the output you get from curl yourself. Using python would look something like

import json
import shlex
from subprocess import Popen, PIPE

command = 'curl -XGET http://localhost:9200'

p = Popen(shlex.split(command), stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate()

if p.returncode != 0:
    print(err)

j = json.loads(output.decode("utf-8"))
print(json.dumps(j, indent=4, sort_keys=True))

However, if what you want long term is to actually connect multiple processes with pipes, well there it depends on the scenario. The easiest method is to pass shell=True to Popen and pass the exact command (not a list of arguments). This delegates everything to the shell. I need to warn you that this is very exploitable when the command is based off user input. Both 2.x pipes.quote() and 3.x shlex.quote() have a recommendation of how to escape the command so it should be safe.

from subprocess import Popen, PIPE

command = 'curl -XGET http://localhost:9200 | python -m json.tool'

p = Popen(command, stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True)
output, err = p.communicate()

if p.returncode != 0:
    print(err)

print(output.decode("utf-8"))

So if you find yourself needing to connect processes but have something based on user input, you can use multiple processes and connect them yourself.

import shlex
from subprocess import Popen, PIPE

command1 = 'curl -XGET http://localhost:9200'
command2 = 'python -m json.tool'

p1 = Popen(shlex.split(command1), stdout=PIPE)
p2 = Popen(shlex.split(command2), stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()
output, err = p2.communicate()

if p2.returncode != 0:
    print(err)

print(output.decode("utf-8"))

This question has a bunch more on the topic if you are curious.

Cant use python subprocess to get return value from curl call

1 Answers