Running Batch python processes on Google Cloud

Question

I have couple of Python scripts which I would like to schedule to run once a month on Google cloud. The scripts basically trigger DLP jobs, extract data catalog information to a file in GCS. These batch workloads would hardly run for 30 mins. And so I don't need to use services like GKE, composer etc which are very resource intensive.

For these batch workloads I would like to know the best options available in GCP. Looking at some of the blog posts I found below article to use Cloud Scheduler-> Pub/Sub-> Cloud Functions -> Create VM (using a startup script). https://medium.com/google-cloud/running-a-serverless-batch-workload-on-gcp-with-cloud-scheduler-cloud-functions-and-compute-86c2bd573f25

I have below questions with above design.. 1) How long does the Cloud Function run as it starts the VM? I know cloud function has a timeout of 9mins ..what happens if the VM takes longer than 9mins to process the startup script?

Any other design ideas are much appreciated.

Thanks

You can start a VM instance asynchronously. Your function does not need to wait for the VM to run. — John Hanley

mesmacosta mesmacosta · Accepted Answer · 2020-05-12T21:00:08

I'm the author of that medium post.

1) How long does the Cloud Function run as it starts the VM? You can change the Cloud Function code to not wait for the response, It's using NodeJS so you just don't have to wait for the Promise. Also in that solution the Cloud Function job is only to trigger the VM creation.

  .createVM(vmName, vmConfig)
  .then(data => {
    // Operation pending.
    const vm = data[0];
    const operation = data[1];
    console.log(`VM being created: ${vm.id}`);
    console.log(`Operation info: ${operation.id}`);
    return operation.promise();
    // This will return right away with the VM pending state, you can finish 
    // your logic here, and not wait for VM creation to finish.
    // You can even ignore this step if you don't need the VM ID logged for 
    // debugging purposes
  })
  .then(() => {
    const message = 'VM created with success, Cloud Function finished execution.';
    console.log(message);
  }

Using that same code, in the worst case (if it takes more than 9 minutes), the Cloud Function will timeout but the VM creation will continue.

Running Batch python processes on Google Cloud

2 Answers