python - requests: disable auto decoding

Question

Can you disable the auto decoding feature in requests version 1.2.3?

I've looked through the documentation and couldn't find anything, I'm currently experiencing a gzip decode error and want to manually debug the data coming through the request.

Answer 1

You can access the raw response like this:

resp = requests.get(url, stream=True)
resp.raw.read()

In order to use raw you need to set stream=True for the original request. Also, raw is a file-like object, and reading from response.content will affect the seek cursor. In other words: If you already (tried to) read response.content, response.raw.read() will return an empty string.

See FAQ: Encoded Data and Quickstart: raw response content in the requests documentation.

Answer 2

import requests

r = requests.get(url, stream=True)
with open(local_filename, 'wb') as f:
    for chunk in r.raw.stream(1024, decode_content=False):
        if chunk:
            f.write(chunk)

This way, you will avoid automatic decompress of gzip-encoded response, and still write it to file chunk by chunk (useful for getting big files)