python request ssh tunnel as proxy

Question

Is it possible to use python requests with ssh tunnel as proxy ? How can I achieve that ? Tried already this but no success :

from sshtunnel import SSHTunnelForwarder
import requests as r
from bs4 import BeautifulSoup as soup

server = SSHTunnelForwarder(
        'sship',
        ssh_username="XXXX",
        ssh_password="XXXX",
        remote_bind_address=('127.0.0.1', 8080)
    )

server.start()

print(server.local_bind_port)


proxies = {
    "http": "http://127.0.0.1:8080",
    }

url = 'http://www.google.com'
headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': '[email protected]'  # This is another valid field
}

data = r.get(url, headers=headers , proxies = proxies)
page_data = soup(data.text , 'html.parser')

print page_data

this is the error that I get:

37657 Traceback (most recent call last): File "ssh.py", line 27, in data = r.get(url, headers=headers , proxies = proxies) File "/usr/lib/python2.7/site-packages/requests-2.19.1-py2.7.egg/requests/api.py", line 72, in get return request('get', url, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.19.1-py2.7.egg/requests/api.py", line 58, in request return session.request(method=method, url=url, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.19.1-py2.7.egg/requests/sessions.py", line 512, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.19.1-py2.7.egg/requests/sessions.py", line 622, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests-2.19.1-py2.7.egg/requests/adapters.py", line 507, in send raise ProxyError(e, request=request) requests.exceptions.ProxyError: HTTPConnectionPool(host='127.0.0.1', port=80): Max retries exceeded with url: http://google.com/ (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)))

Matt Calderaz Matt Calderaz · Accepted Answer · 2019-03-21T21:21:52

I think you need to either explicitly set the local bind address in SSHTunnel, or set up the port for your proxy like this:

`
https_tunnel.start()
    proxies = {
        #'http': 'http://localhost:{}'.format(str(http_tunnel.local_bind_port)),
        'https': 'https://localhost:{}'.format(str(https_tunnel.local_bind_port))
    }
`

where https_tunnel is an instance of SSHTunnelForwarder. I've done that, however I get errors around actually sending the GET request:

python2.7/site-packages/requests/adapters.py", line 490, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",))

python request ssh tunnel as proxy

1 Answers