JSON encoding and downloading from a URL

Question

I'm trying to get JSON from a Steam inventory. I get the data like this :

def downloadString(url):
    req = urllib.request.Request(
        url, 
        data=None, 
        headers={
            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
        }
    )
    f = urllib.request.urlopen(req)
    return f.read().decode("utf-8")

Now, I get a problem with encoding. Steam is using symbols like "Black star" ('\u2605') which causes the json part to crash :

def test(string):
    print(json.loads(string))

test(downloadString(url))

File "C:\Python34\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\u2605' in position 83559: character maps to

I don't really understand how I can do something else... Any help would be... helpful.

Already tried : File "C:\Python34\lib\json_init_.py", line 312, in loads s.__class__.__name__)) TypeError: the JSON object must be str, not 'bytes' — Thomas Kowalski
have you tried using requests library -> requests.get(url).json() ? — YOBA

Jesse Shapiro Jesse Shapiro · Accepted Answer · 2016-06-01T20:33:13

JSON can be encoded in any of UTF-8, UTF-16, or UTF-32. I would try decoding as UTF-16 instead of UTF-8.

JSON encoding and downloading from a URL

1 Answers