Advise needed - Running Python code on GOOGLE CLOUD PLATFORM serverless

Question

I have a python code which reads data from one cloud system via rest api using the requests module and then writes data back to another cloud system via rest api . This code runs anywhere from 1 to 4 hours every week. Is there a place in Google Cloud Platform , I can execute this code on a periodic basis. Sort of like a scheduled batch job . Is there a serverless option to do this in App Engine . I know about the App engine cron service but seems like it is only for calling a URL regularly . Any thoughts ? Appreciate your help.

Answer 1

Google Cloud Scheduler could be the tool you are looking for. As it is mentioned in its documentation:

Cloud Scheduler is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations, and more. You can automate everything, including retries in case of failure to reduce manual toil and intervention.

Here you have the quickstart for Cloud Scheduler, and also another tutorial for Cron jobs.

Answer 2

You can use the Google Genomics API pipelines.run endpoint to run a long-running job on a Google Compute Engine virtual machine and then it will destroy the machine when it's done. If your job will run for less than 24 hours and it can handle a failure, then you can use a Preemptible VM to save cost.

Pipelines: Run

https://cloud.google.com/genomics/reference/rest/v2alpha1/pipelines/run

Preemptible Virtual Machines

https://cloud.google.com/preemptible-vms/

You could use Cloud Scheduler to kick off the job

Pipelines may be preferred to trying to use one of the serverless technologies because they don't tend to handle the long running jobs as well.

Answer 3

You can use AI Platform Training to run any arbitrary Python package — it doesn’t have to be a machine learning job.