HTTPException: Deadline exceeded while waiting for HTTP response from URL

1
votes

I have this code at the top of my Google App Engine program:

from google.appengine.api import urlfetch
urlfetch.set_default_fetch_deadline(60)

I am using an opener to load stuff:

    cj = CookieJar()        

    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor( cj ) )
    opener.addheaders = [ ( 'User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64)' ) ]

    resp = opener.open( 'http://www.example.com/' )

an exception is being thrown after 5 seconds: 

 File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 1222, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/urllib2.py", line 1187, in do_open
    r = h.getresponse(buffering=True)
  File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/gae_override/httplib.py", line 524, in getresponse
    raise HTTPException(str(e))
HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://www.example.com

How can I avoid the error?

2
pythongoogle-app-engineurllib2urlfetch
Another way to do it if you can't do it within age's time constraint for an http request, is to make the job a task or a mapreduce job. These 2 ways have more flexible time limts for methods and http request. Regular gae http request have a 15-20 second timeout limit so if you need more time-consuming jobs there are other alternatives. – Niklas R.

2 Answers

1
votes

Did you try setting the timeout on the .open() call?

resp = opener.open('http://example.com', None, 60)

If you reach the timeout as specified by set_default_fetch_deadline, Python will throw a DownloadError or DeadlineExceededErrors exception: https://cloud.google.com/appengine/docs/python/urlfetch/exceptions

0
votes

You can also patch the httplib2 library and set the deadline to 60 seconds

httplib2/__init__.py:
      def fixed_fetch(url, payload=None, method="GET", headers={},
                      allow_truncated=False, follow_redirects=True,
                      deadline=60):
          return fetch(url, payload=payload, method=method, headers=headers,
                       allow_truncated=allow_truncated,
                       follow_redirects=follow_redirects, deadline=60,
                       validate_certificate=validate_certificate)
      return fixed_fetch

It is a work around.