openssl, python requests error: "certificate verify failed"

Question

If I run the following command from my development box:

$ openssl s_client -connect github.com:443

I get the following last line of output:

Verify return code: 20 (unable to get local issuer certificate)

If I try to do this with requests I get another failed request:

>>> import requests
>>> r = requests.get('https://github.com/', verify=True)

With an exception raised:

SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I can also run the first command with the verify flag and get similar output:

$ openssl s_client -connect github.com:443 -verify 9
...
Verify return code: 27 (certificate not trusted)

Basically this is telling me that there is a problem with the certificates. I can specify a specific certificate with both methods and it will work:

$ openssl s_client -connect github.com:443 -CAfile /etc/ssl/certs/DigiCert_High_Assurance_EV_Root_CA.pem -verify 9
...
Verify return code: 0 (ok)

and:

>>> r = requests.get('https://github.com/', verify='/etc/ssl/certs/DigiCert...pem')
<Response [200]>

So, to my question, what exactly is wrong here? Shouldn't requests/openssl already know where to find valid certs?

Other Info:

Python==2.7.6
requests==2.2.1
openssl 0.9.8h

Also, I know passing verify=False to the requests.get method will work too, but I do want to verify.

EDIT

I've confirmed that, as @Heikki Toivonen indicated in an answer, specifying the -CAfile flag for the version of openssl that I'm running works.

$ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`
...
Verify return code: 0 (ok)

So there is nothing wrong with the version of openssl that I'm running, and there is nothing wrong with the default cacert.pem file that requests provides.

Now that I know openssl is meant to work that way, that the CAfile or the place to find certs has to be specified, I'm more concerned about getting requests to work.

If I run:

>>> r = requests.get('https://github.com/', verify='path to cacert.pem file')

I'm still getting the same error as before. I even tried downloading the cacert.pem file from http://curl.haxx.se/ca and it still didn't work. requests only seems to work (on this specific machine) if I specify a specific vendor cert file.

A side note: On my local machine everything is working as expected. There are several difference between the two machines though. I so far haven't been able to determine what the specific difference is that causes this issue.

jww jww · Accepted Answer · 2014-02-26T05:32:27

If I run the following command from my development box:
$ openssl s_client -connect github.com:443
I get the following last line of output:
Verify return code: 20 (unable to get local issuer certificate)

You are missing DigiCert High Assurance EV CA-1 as a root of trust:

$ openssl s_client -connect github.com:443
CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
...
Start Time: 1393392088
Timeout   : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)

Download DigiCert High Assurance EV CA-1 from DigiCert Trusted Root Authority Certificates:

$ wget https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt
--2014-02-26 00:27:50--  https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt
Resolving www.digicert.com (www.digicert.com)... 64.78.193.234
...

Convert the DER encoded certifcate to PEM:

$ openssl x509 -in DigiCertHighAssuranceEVCA-1.crt -inform DER -out DigiCertHighAssuranceEVCA-1.pem -outform PEM

Then, use it with OpenSSL via the -CAfile:

$ openssl s_client -CAfile DigiCertHighAssuranceEVCA-1.pem -connect github.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1
verify return:1
depth=0 businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Delaware, serialNumber = 5157550, street = 548 4th Street, postalCode = 94107, C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
verify return:1
---
Certificate chain
 0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHOjCCBiKgAwIBAgIQBH++LkveAITSyvjj7P5wWDANBgkqhkiG9w0BAQUFADBp
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSgwJgYDVQQDEx9EaWdpQ2VydCBIaWdoIEFzc3VyYW5j
ZSBFViBDQS0xMB4XDTEzMDYxMDAwMDAwMFoXDTE1MDkwMjEyMDAwMFowgfAxHTAb
BgNVBA8MFFByaXZhdGUgT3JnYW5pemF0aW9uMRMwEQYLKwYBBAGCNzwCAQMTAlVT
MRkwFwYLKwYBBAGCNzwCAQITCERlbGF3YXJlMRAwDgYDVQQFEwc1MTU3NTUwMRcw
FQYDVQQJEw41NDggNHRoIFN0cmVldDEOMAwGA1UEERMFOTQxMDcxCzAJBgNVBAYT
AlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xEzARBgNVBAMTCmdpdGh1Yi5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDt04nDXXByCfMzTxpydNm2WpVQ
u2hhn/f7Hxnh2gQxrxV8Gn/5c68d5UMrVgkARWlK6MRb38J3UlEZW9Er2TllNqAy
GRxBc/sysj2fmOyCWws3ZDkstxCDcs3w6iRL+tmULsOFFTmpOvaI2vQniaaVT4Si
N058JXg6yYNtAheVeH1HqFWD7hPIGRqzPPFf/jsC4YX7EWarCV2fTEPwxyReKXIo
ztR1aE8kcimuOSj8341PTYNzdAxvEZun3WLe/+LrF+b/DL/ALTE71lmi8t2HSkh7
bTMRFE00nzI49sgZnfG2PcVG71ELisYz7UhhxB0XG718tmfpOc+lUoAK9OrNAgMB
AAGjggNUMIIDUDAfBgNVHSMEGDAWgBRMWMsl8EFPUvQoyIFDm6aooOaS5TAdBgNV
HQ4EFgQUh9GPGW7kh29TjHeRB1Dfo79VRyAwJQYDVR0RBB4wHIIKZ2l0aHViLmNv
bYIOd3d3LmdpdGh1Yi5jb20wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjBjBgNVHR8EXDBaMCugKaAnhiVodHRwOi8vY3JsMy5k
aWdpY2VydC5jb20vZXZjYTEtZzIuY3JsMCugKaAnhiVodHRwOi8vY3JsNC5kaWdp
Y2VydC5jb20vZXZjYTEtZzIuY3JsMIIBxAYDVR0gBIIBuzCCAbcwggGzBglghkgB
hv1sAgEwggGkMDoGCCsGAQUFBwIBFi5odHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9z
c2wtY3BzLXJlcG9zaXRvcnkuaHRtMIIBZAYIKwYBBQUHAgIwggFWHoIBUgBBAG4A
eQAgAHUAcwBlACAAbwBmACAAdABoAGkAcwAgAEMAZQByAHQAaQBmAGkAYwBhAHQA
ZQAgAGMAbwBuAHMAdABpAHQAdQB0AGUAcwAgAGEAYwBjAGUAcAB0AGEAbgBjAGUA
IABvAGYAIAB0AGgAZQAgAEQAaQBnAGkAQwBlAHIAdAAgAEMAUAAvAEMAUABTACAA
YQBuAGQAIAB0AGgAZQAgAFIAZQBsAHkAaQBuAGcAIABQAGEAcgB0AHkAIABBAGcA
cgBlAGUAbQBlAG4AdAAgAHcAaABpAGMAaAAgAGwAaQBtAGkAdAAgAGwAaQBhAGIA
aQBsAGkAdAB5ACAAYQBuAGQAIABhAHIAZQAgAGkAbgBjAG8AcgBwAG8AcgBhAHQA
ZQBkACAAaABlAHIAZQBpAG4AIABiAHkAIAByAGUAZgBlAHIAZQBuAGMAZQAuMH0G
CCsGAQUFBwEBBHEwbzAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQu
Y29tMEcGCCsGAQUFBzAChjtodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGln
aUNlcnRIaWdoQXNzdXJhbmNlRVZDQS0xLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqG
SIb3DQEBBQUAA4IBAQBfFW1nwzrVo94WnEUzJtU9yRZ0NMqHSBsUkG31q0eGufW4
4wFFZWjuqRJ1n3Ym7xF8fTjP3fdKGQnxIHKSsE0nuuh/XbQX5DpBJknHdGFoLwY8
xZ9JPI57vgvzLo8+fwHyZp3Vm/o5IYLEQViSo+nlOSUQ8YAVqu6KcsP/e612UiqS
+UMBmgdx9KPDDzZy4MJZC2hbfUoXj9A54mJN8cuEOPyw3c3yKOcq/h48KzVguQXi
SdJbwfqNIbQ9oJM+YzDjzS62+TCtNSNWzWbwABZCmuQxK0oEOSbTmbhxUF7rND3/
+mx9u8cY//7uAxLWYS5gIZlCbxcf0lkiKSHJB319
-----END CERTIFICATE-----
subject=/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
---
No client certificate CA names sent
---
SSL handshake has read 4139 bytes and written 446 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 59D2883BBCE8E81E63E5551FAE7D1ACC00C49A9473C1618237BBBB0DD9016B8D
    Session-ID-ctx: 
    Master-Key: B6D2763FF29E77C67AD83296946A4D44CDBA4F37ED6F20BC27602F1B1A2D137FACDEAC862C11279C01095594F9776F79
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1393392673
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

Shouldn't requests/openssl already know where to find valid certs?

No. OpenSSL trusts nothing by default. Its a polar opposite of a browser's model, where nearly everything is trusted by default.

 $ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`
 ...
 >>> r = requests.get('https://github.com/', verify='path to cacert.pem file')

Why would you trust hundreds of CAs and subordinate CAs (re: cacert.pem) when you know the one CA that is certifying the public key for the site? Trust the one required root and nothing more: DigiCert High Assurance EV CA-1.

Trusting everything - as in the browser's model - is what allowed Comodo Hacker to spoof certificates for Gmail, Hotmail, Yahoo, etc when the Diginotar root was compromised.

openssl, python requests error: "certificate verify failed"

3 Answers