HTTP Post using urllib vs requests in Python

Question

I'm trying to use web api which requires Post field in HTTP (the web is https) with python3.6.2. I have tried urllib.request.urlopen but the response was

urllib.error.HTTPError: HTTP Error 403: Forbidden

Here is the code I used:

from urllib.parse import urlencode
from urllib.request import Request, urlopen
from hashlib import sha256
import time

key = 'xxxxxxxxxx'
secret_key = 'yyyyyyyyyyyy'
nonce = int(time.time())

signature = sha256((key + str(nonce) + secret_key).encode()).hexdigest()

url = "https://xxxxx/api/"
post_fields = {'key': key, 'nonce': nonce,
                'signature': signature}

request = Request(url, data=urlencode(post_fields).encode())
response = urlopen(request).read().decode()

However I have tried requests library with this code:

import requests
from hashlib import sha256
import time

key = 'xxxxxxxxxx'
secret_key = 'yyyyyyyyyyyy'
nonce = int(time.time())

signature = sha256((key + str(nonce) + secret_key).encode()).hexdigest()

url = "https://xxxxx/api/"
post_fields = {'key': key, 'nonce': nonce,
                        'signature': signature}
response = requests.post(url, post_fields)

and it works. I really want to know the different since urllib.request can send post request when data parameter is specified.

Martijn Pieters Martijn Pieters · Accepted Answer · 2017-08-23T07:19:02

Both snippets of code send a correct POST request. The only difference between them will be what headers are sent. Apart from the headers that are exactly the same, urllib.request will send:

Accept-Encoding: identity
User-Agent: Python-urllib/3.6

while requests will send:

Accept: */*
Accept-Encoding: gzip, deflate
User-Agent: python-requests/2.18.1

You'd have to experiment with adding and altering headers to your urllib.request code to see if those matter.

You can always use http://httpbin.org/post as the URL to see what information you posted; that service echoes back what it receives as a JSON object. See http://httpbin.org for more information.

HTTP Post using urllib vs requests in Python

1 Answers