App Engine: Retrieving Large Files from Third Party Site

0
votes

In my Google App Engine application, a task has to download a large file from a third party site, process it, and store the results in the datastore/blobstore. Given that urlfetch can only receive 32MB of data in a response, and this file is larger than 32MB, what is the best way of doing this?

The only solution I can think of is to build another server not subject to the urlfetch limits and then have the server download the large file and upload it to a GAE blobstore upload url. This seems like a complete headache to implement, and having a second server defeats the purpose of using GAE. Isn't there an easier way of getting large files into Google App Engine?

1
google-app-engineurlfetch
You can download the file in parts, 32mb per part. But you also need to store the download data somewhere, because there is also a limiti of data you can store in memory. You can use the 'Range' HTTP Header (but the server where you download the file has to support this header) - Deviling Master

1 Answers

0
votes

You can try the new sockets api: https://developers.google.com/appengine/docs/python/sockets/overview#limitations-and-restrictions

It says if you want to get around the request limit you need to upload your own httplib. I can't think of reason why this won't work yet, run it in a backends to avoid timeout and writing chunks of data as you recieve it to google cloud storage or blobstore (but its files api is depracated).

But other than that yeah there is no other way than what Deviling Master said and your another server that handles it.