Posting data using urllib without SSL certificate validation

Question

I need to post data to a REST interface, however the receiving host is using a self-signed certificate (and this is not about to change) so I need to ignore the obvious certificate validation error that I receive.

My initial script looked like this:

from urllib.request import Request, urlopen
from urllib.parse import urlencode

post_url = "https://myserver.mydomain.com:30005/myapipath"
post_payload = { "event": { "Title": "Something, sometime, something, Python"} }
post_headers = {'Content-Type': 'application/xml'}

omi_post = Request(url=post_url, data=post_payload, headers=post_headers)

urlopen(omi_post)

And as mentioned, this generates the following stacktrace:

Traceback (most recent call last): File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1026, in _send_output self.send(msg) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 964, in send self.connect() File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1400, in connect server_hostname=server_hostname) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\ssl.py", line 407, in wrap_socket _context=self, _session=session) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\ssl.py", line 814, in init self.do_handshake() File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\ssl.py", line 1068, in do_handshake self._sslobj.do_handshake() File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "send_data.py", line 16, in urlopen(post_it_already) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 223, in urlopen return opener.open(url, data, timeout) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 526, in open response = self._open(req, data) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 544, in _open '_open', req) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 504, in _call_chain result = func(*args) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1361, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1320, in do_open raise URLError(err) urllib.error.URLError:

So I found another post that talked about adding in an SSL context, which I did as follows:

from urllib.request import Request, urlopen
from urllib.parse import urlencode
import ssl

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

post_url = "https://myserver.mydomain.com:30005/myapipath"
post_payload = { "event": { "Title": "Something, sometime, something, Python"} }
post_headers = {'Content-Type': 'application/xml'}

post_it_already = Request(url=post_url, data=post_payload, headers=post_headers)

urlopen(post_it_already, context=ctx)

However this generates the following stacktrace (no matter how much I shake my fist at it):

Traceback (most recent call last): File "send_data.py", line 15, in urlopen(post_it_already, context=ctx) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 223, in urlopen return opener.open(url, data, timeout) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 526, in open response = self._open(req, data) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 544, in _open '_open', req) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 504, in _call_chain result = func(*args) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1361, in https_open context=self._context, check_hostname=self._check_hostname) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "C:\Users\myusername\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1064, in _send_output + b'\r\n' TypeError: can't concat str to bytes

I'm at a loss as to why it's complaining about concatenating a string with a bytes object. I assume it doesn't like my string type URL and my data type payload and headers? However I'm unsure what to do next.

Or is this a classic case of someone taking 2 separate sample codes and expecting them to work together when they are in fact like peanuts and bubble gum?

Answer 1

Your post_payload should be of type 'bytes' rather than 'str'.

From the Python docs:

class urllib.request.Request ...

...

For an HTTP POST request method, data should be a buffer in the standard application/x-www-form-urlencoded format. The urllib.parse.urlencode() function takes a mapping or sequence of 2-tuples and returns an ASCII string in this format. It should be encoded to bytes before being used as the data parameter.

...

You can convert post_payload to bytes using the .encode() method as detailed in this Python urllib howto.

This is separate to the SSL validation issue, but presumably because it attempted to POST data, the HTTPS connection was successfully established.

Answer 2

You can use @contextlib.contextmanager decorator with requests module instead. It's a much simpler approach.

The below code works just fine

@contextlib.contextmanager
def no_ssl_verification():
    old_request = requests.Session.request
    requests.Session.request = partialmethod(old_request,verify=False)

    warnings.filterwarnings('ignore', 'Unverified HTTPS request')
    yield

    warnings.resetwarnings()
    requests.Session.request = old_request

What the above code does is it generates a partial method for requests.Session.request with 'verify=False' and filters any warnings that say 'Unverified Https request'.

with no_ssl_verification:
   request = requests.post( 
                           url,
                           json={
                                    'name' : some_name
                                },auth(username,password)
                           )

Hope it helps, please let me know in case you find a better solution.