How to prevent Python request from aborting after running

Question

I'm running a python request to

url = 'https://www.walmart.com/store/1003-York-pa/search?query=ice%20cream'
api_url = 'https://www.walmart.com/store/electrode/api/search'

    params = {
        'query': word,
        'cat_id': 0,
        'ps': 24,
        'offset': 0,
        'prg': 'desktop',
        'stores': re.search(r'store/(\d+)', url).group(1)
    }

    data = requests.get(api_url, params=params).json()

I am trying to loop through thousands of query word. After running through couple of thousand of query request using the code the connection aborts and display this error message

data = requests.get(api_url, params=params).json() File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py", line 72, in get return request('get', url, params=params, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py", line 58, in request return session.request(method=method, url=url, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 512, in request resp = self.send(prep, **send_kwargs) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 622, in send r = adapter.send(request, **kwargs) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/adapters.py", line 495, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))

what is causing it to abort and how can I fix it?

What do you mean "this works fine connect aborts after running for sometime and produce this error statements"? How can it "work fine" but also "abort after running for sometime"? — CryptoFool
I just tried this code, and it worked fine for me, including exiting right away with an exit code of 0 and no exception. Nothing at all printed to the console. I got back a ton of JSON. — CryptoFool
It sounds like the code is probably fine. I suspect it is either the server is rejecting you after a time, or a network connection problem. — sniperd
"connection reset by peer" means the remote server is disconnecting you. The way you handle that is generally by setting up a loop and a try/except block around your request so that you can retry on failure, potentially with some sort of backoff to avoid too many rapid requests. — larsks

Fabiano Teichmann Fabiano Teichmann · Accepted Answer · 2020-09-20T01:44:17

It would appear that ECONNRESET means that the other side has closed the connection without reading.My suggestion for you is catch payload sender when the error occurred. And try again send payload. If you succeed in that case it may be instability in your network in prod.

How to prevent Python request from aborting after running

1 Answers