'set' object has no attribute 'setdefault'. Error in scraping data using using Requests

0
votes

I am trying to hit a website using python Requests, but it's giving me error. 

import requests 
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'}
URL = ""
PROXY = {'https://surfproxy.de.db.com:8080' }
response = requests.get(URL , proxies = PROXY, headers: headers)

The error logs:

File "", line 1, in runfile('C:/Users/vermanjb/JiraScrapping.py', wdir='C:/Users/vermanjb')

File "C:\Program Files\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile execfile(filename, namespace)

File "C:\Program Files\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 88, in execfile exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

File "C:/Users/vermanjb/JiraScrapping.py", line 12, in response = requests.get(URL , proxies = PROXY)

File "C:\Program Files\Anaconda3\lib\site-packages\requests\api.py", line 67, in get return request('get', url, params=params, **kwargs)

File "C:\Program Files\Anaconda3\lib\site-packages\requests\api.py", line 53, in request return session.request(method=method, url=url, **kwargs)

File "C:\Program Files\Anaconda3\lib\site-packages\requests\sessions.py", line 459, in request prep.url, proxies, stream, verify, cert

File "C:\Program Files\Anaconda3\lib\site-packages\requests\sessions.py", line 619, in merge_environment_settings proxies.setdefault(k, v)

AttributeError: 'set' object has no attribute 'setdefault'

1
pythonbeautifulsouppython-requests
Why are you using a set as the value of proxies? – user2357112 supports Monica
PROXY needs to be a dictionary not a set – Jean-François Fabre♦
Change to PROXY = {'https': https://surfproxy.de.db.com:8080' }. Also headers: headers? Shouldn't this be headers=headers? – RoadRunner
Thanks. But i'm getting aniother error. ConnectionError: HTTPSConnectionPool(host='', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 502 notresolvable',))) But i'm able to ping this website from CMD using the same proxy – user8569142
this is a totally different issue, and it's a system issue not a programming one – bruno desthuilliers

1 Answers

0
votes

There are at least two problems you should fix:

  1. The proxies parameter should be a dict, not a set.
  2. You have a syntax error in your headers parameter.

Try this instead:

headers = {'User-Agent': 'Mozilla/5.0...'}
url = 'http://www.yoursite.com/'
proxies = {
    'http': 'http://surfproxy.de.db.com:8080',
    'https': 'http://surfproxy.de.db.com:8080'
}
response = requests.get(url , proxies=proxies, headers=headers)