Long running python process with Google Cloud Functions

Question

I'm trying to run a python script in Google Cloud which will download 50GB of data once a day to a storage bucket. That download might take longer than the timeout limit on the Google Cloud Functions which is set to 9 minutes. The request to invoke the python function is triggered by HTTP.

Is there a way around this problem ? I don't need to run a HTTP Restful service as this is called once a day from an external source. (Can't be scheduled) .

The whole premise is do download the big chuck of data directly to the cloud. Thanks for any suggestions.

Answer 1

9 minutes is a hard limit for Cloud Functions that can't be exceeded. If you can't split up your work into smaller units, one for each function invocation, consider using a different product. Cloud Run limits to 15 minutes, and Compute Engine has no limit that would apply to you.

Answer 2

Google Cloud Scheduler may work well for that.

Here is a nice google blog post that shows example of how to set up a python script.

p.s. you would probably want to connect it to the App Engine for the actual execution.