python - urllib2 opener providing wrong charset

Question

When I open the url and read it, I can't recognize it. But when I check the content header it says it is encoded as utf-8. So I tried to convert it to unicode and it complained UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) using unicode().

.encode("utf-8") produces UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128)

.decode("utf-8") produced UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte.

I have tried everything I can come up with(I'm not that good at encodings)

I would be happy if I could get this to work. Thanks.

HTTP headers can be wrong - look for a meta tag in the HTML itself specifying the HTML. It might be Latin-1 (.decode('latin-1')). — Thomas K
Tried that. And I'm quite sure it's utf-8 since chrome sais so when I visit the site. — thabubble
If your stream is really utf8-encoded, .decode("utf-8") will work. So, you must be mistaken about something. Can you post the URL (or: a URL) that's giving you trouble? — alexis
Could you create a minimal complete example that shows the problem e.g., print-page-info.py — jfs

Vanuan Vanuan · Accepted Answer · 2012-11-20T23:25:00

This is a common mistake. The server sends gzipped stream.

You should unpack it first:

response = opener.open(self.__url, data)
if response.info().get('Content-Encoding') == 'gzip':
    buf = StringIO.StringIO( response.read())
    gzip_f = gzip.GzipFile(fileobj=buf)
    content = gzip_f.read()
else:
    content = response.read()

python - urllib2 opener providing wrong charset

2 Answers