Updating Python code for signing an Amazon API request to work with Python 3

Question

I have some working code that gets me some sales data from the Amazon API. It works in python 2.7, but I'm having trouble updating it to python 3.6 The error comes from signing the request. My code is as follows:

import base64, hashlib, hmac, urllib
from time import gmtime, strftime
from requests import request
import xml.etree.ElementTree as ET

def get_timestamp():
    """Return correctly formatted timestamp"""
    return strftime("%Y-%m-%dT%H:%M:%SZ", gmtime())

def calc_signature(method, domain, URI, request_description, key):
    """Calculate signature to send with request"""
    sig_data = method + '\n' + \
        domain.lower() + '\n' + \
        URI + '\n' + \
        request_description

    hmac_obj = hmac.new(key, sig_data, hashlib.sha256)
    digest = hmac_obj.digest()

    return  urllib.parse.quote(base64.b64encode(digest), safe='-_+=/.~')

SECRET_KEY = 'xxxxxxxxxxxxxxxxxxxxx'
AWS_ACCESS_KEY = 'xxxxxxxxxxxxxxxxxxx'
SELLER_ID = 'xxxxxxxxxxxxxxxxxx'
MARKETPLACE_ID = marketplace_id
version = '2013-09-01'

Action = 'ListOrders'
SignatureMethod = 'HmacSHA256'
SignatureVersion = '2'
Timestamp = get_timestamp()
Version = '2013-09-01'
CreatedAfter = '2017-05-26T23:00:57Z'
URI = '/Orders/2013-09-01'
domain = 'mws.amazonservices.com'
proto = 'https://'
method = 'POST'

payload = {
    'AWSAccessKeyId': AWS_ACCESS_KEY,
    'Action': Action,
    'SellerId': SELLER_ID,
    'SignatureVersion': SignatureVersion,
    'Timestamp': Timestamp,
    'Version': Version,
    'SignatureMethod': SignatureMethod,
    'CreatedAfter': CreatedAfter,
    'MarketplaceId.Id.1': MARKETPLACE_ID
}

request_description = '&'.join(['%s=%s' % (k, urllib.parse.quote(payload[k], safe='-_.~').encode('utf-8')) for k in sorted(payload)])

sig = calc_signature(method, domain, URI, request_description, SECRET_KEY)

url = '%s%s?%s&Signature=%s' % \
    (proto+domain, URI, request_description, urllib.parse.quote(sig))

headers = {
    'Host': domain,
    'Content-Type': 'text/xml',
    'x-amazon-user-agent': 'python-requests/1.2.0 (Language=Python)'
}

r = request(method, url, headers=headers)

print(r.status_code)

print(r.text)

The error is thrown by the calc_signature method (again, this worked in python 2.7) which tells me:

TypeError: key: expected bytes or bytearray, but got 'str'

After doing some digging, I was able to correct that by adding .encode('utf-8') to the line:

hmac_obj = hmac.new(key.encode('utf-8'), sig_data.encode('utf-8'), hashlib.sha256)

AFter applying the encoding to the inputs for hmac.new, the code now runs, however, Amazon rejects the request and says:

<Error>
    <Type>Sender</Type>
    <Code>InvalidParameterValue</Code>
    <Message>Value b&apos;2&apos; for parameter SignatureVersion is invalid.
    </Message>
</Error>

I have not been able to find out what has changed with the hmac module between python versions that is causing it to calculate an incorrect signature.

The solution to that question is something I've tried (if you read the question) - Maybe I'm applying it wrong? I get the signature to calculate, but it must be calculating incorrectly because Amazon rejects it. — Casey
Try using an integer for the SignatureVersion. The response from AWS uses HTML entities and reads as follows: b'2' -> b'2'. So it looks like that at some point the unicode string "2" is encoded and the result is cast back to unicode which results in "b'2'" (str('2'.encode('ascii')) == "b'2'"). EDIT: In order to work with arbitrary version numbers one has to use a string, so this can't be the ultimate solution. Not sure at what point this conversion happens but it should decode instead of cast to str. — a_guest

a_guest a_guest · Accepted Answer · 2017-06-27T13:31:06

Not adding this as a comment because it might exceed the character limit.

The problem is the following line:

request_description = '&'.join(
    ['%s=%s' % (
        k,
        urllib.parse.quote(payload[k], safe='-_.~').encode('utf-8')
     )
     for k in sorted(payload)]
)

You are encoding the header values and then they are cast back to str during the string interpolation ('%s=%s' % (...)). For the SignatureVersion for example this results in

>>> str('2'.encode('ascii'))
"b'2'"
>>> '%s' % '2'.encode('ascii')
"b'2'"

which is what AWS reports (using HTML entities; ' represents ').

I'm not sure what you are trying to achieve with this line. You can encode the payload via

urllib.parse.urlencode(payload).encode('ASCII')

(Referring to this answer.)

Updating Python code for signing an Amazon API request to work with Python 3

1 Answers