How to periodically train and deploy new Machine Learning models on Google Cloud ML? How to automate this process?

Question

I am successful in training models on GCP MLE, but now as new data is generated the model needs to be updated every couple of weeks and I am doing this manually, can someone kindly help me to automate this process.

My current architecture is I have data as csv files stored on GCS buckets, I manually run a script and train the model and export the newly trained model as webservice (via cloud endpoints) so users can query new data and get inference...

I would like to build a system where, every 2 weeks i will provide new csv files and a training job should be automatically triggered, and the new model should replace the old model for inference (I need everything to be automated)

Answer 1

You can use Cloud Composer to create a pipeline that runs periodically.

You can watch this talk that explains how to do this for scikit learn models on Cloud ML Engine, but it's the same for TensorFlow. The notebook with the code can be found here.