Python requests connect to blocked server

1
votes

I want to connect to one URL, which is blocked, using python requests.

I used many different proxy servers none worked for me. https://free-proxy-list.net/

import requests

x = '46.21.153.16:3128'
http_proxy  = "http://"+x
https_proxy = "https://"+x

proxyDict = {
  "http": http_proxy,
  "https": https_proxy
}


a = requests.get('https://example.com', proxies=proxyDict)
print(a.status_code)

Traceback (most recent call last): File "t.py", line 14, in a = requests.get('https://my-url', proxies=proxyDict) File "/home/user/Documents/backend1/venv-livedx/lib/python3.5/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/home/user/Documents/livedx/backend1/venv-livedx/lib/python3.5/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/home/user/Documents/backend1/venv-livedx/lib/python3.5/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/home/user/Documents/backend1/venv-livedx/lib/python3.5/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/home/user/Documents/backend1/venv-livedx/lib/python3.5/site-packages/requests/adapters.py", line 510, in send raise ProxyError(e, request=request) requests.exceptions.ProxyError: HTTPSConnectionPool(host='my-url', port=443): Max retries exceeded with url: /management (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',)))

2
pythonpython-3.xvpn
Can browse that url using your browser? – Debendra
yes using VPN we can browse the URL – rahul.m
Try my solution below, and make sure your vpn is connected globally not just in browser. – Debendra

2 Answers

2
votes

Popular network firewall block popular user-agent considering them as bots.

Copy your browser agent and simply try changing the user agent on request. I bet request uses its own default user agent or the python urllib one.

Example:

a = requests.get('https://example.com', headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.152 Safari/537.36'})
1
votes

Try using only the http proxy for the connection. So try changing your code to:

http_proxy  = "http://46.21.153.16:3128"
proxyDict = {
  "http": http_proxy,
  "https": http_proxy
}

The free proxy might not support ssl/tls connection.

Also, since you are using a free proxy make sure it hasn't expired.

Moreover if the url you try to parse has self signed certificates python requests might also fail, so you have to make the request using setting the verify argument to False.

a = requests.get('https://example.com', proxies=proxyDict, verify=False).