How to capture python https traffic in fiddler?

5
votes

Python throws in errors when ever I try to do some data fetching task. This only happens when I set fiddler to decrypt https traffic. I have tried routing python traffic through 127.0.0.1:8888 and same with mozilla inorder to catch its traffic. I also installed the certificate and trusted it via fiddler, I am not sure where I am going wrong.

    raise SSLError(e, request=request)
    requests.exceptions.SSLError: HTTPSConnectionPool(host='google.com', port=443):
    Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFIC
    ATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)'),))

This above is the error I get whenever I try to fetch a page with requests.

2
pythonhttpspython-requestsfiddler
Possible duplicate of How to capture python SSL(HTTPS) connection through fiddler2 - Joe
Already tried the solution, did not work out. - rikoudosenin
plz show us your code. - georgexsh
requests.get("https://google.com"), however code here isnt a problem even pip fails. - rikoudosenin

2 Answers

6
votes

TL;DR The requests library does not use the windows certificate store, it has it's own one (as per https://bugs.python.org/issue28547). This means that your fiddler MITM certificate is not available to python requests by default.

Your options are

  1. Disable SSL verification (verify=False)
  2. Add your cert via the REQUESTS_CA_BUNDLE environment variable
  3. Add your fiddler cert explicitly (verify='\path\to\cert')

More details can be found at http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

On a side note, it does feel a little strange for requests to be using it's own cert bundle, rather than the platform supplied one - especially given all the browsers are happy to use the platform ones.

0
votes

As pointed out by polhemic and Eric Aronesty, for testing purposes, you can set temporarily "CURL_CA_BUNDLE" to an empty string.

import os
os.environ['CURL_CA_BUNDLE'] = ''