Why is urlfetch throwing Download Errors when calling some Google services?

2
votes

I've noticed that some Google services are blocking requests from App Engine servers, resulting in a urlfetch DownloadError. An example would be a feedproxy.google.com url (http://feedproxy.google.com/~r/blabbermouth/~3/cAk78LX4gJE/news.aspx, for example).

This occurs on all the apps I've tried it on, including app IDs I've never used for any kind of url fetching before. This behavior also doesn't occur on the local SDK. This leads me to believe that this is a result of using any GAE IP address when making the request.

The weird thing is that it results in the throwing of a DownloadError, instead of an error status_code in the successfully retrieved response. Using urlfetch or httplib locally works just fine, so this DownloadError I don't yet grok, or it's just a bug, in which case I'll file a ticket.

1
google-app-engineip-addressurlfetch
Someone mentioned (groups.google.com/group/google-appengine/msg/98b38f1f5cdb592c) adding headers={ "Accept-Encoding" : "identity" }, but that didn't seem to help. Maybe you could try setting up a simple server to see what's the difference between the request headers sent by appspot vs. your local machine. Maybe there is some issue here similar to that accept-encoding header issue. - Bemmu
Nope, headers isn't the issue. I've tried the exact same fetch calls locally and on production, and the only difference AFAIK is the IP address being used. - jamtoday

1 Answers

1
votes

Without having a look at your code I will be guessing but since the URL that you are following is a going to redirect are you allowing redirects in your call? Note the follow_redirects=True

e.g. urlfetch.fetch(url, payload=None, method=GET, headers={}, allow_truncated=False, follow_redirects=True, deadline=None)

http://code.google.com/appengine/docs/python/urlfetch/fetchfunction.html