We have a script exporting csv-files from another database and uploading them to a bucket on GCP cloud storage. Now I know there's the possibility to schedule loads into BigQuery using BigQuery Data Transfer Service but I am a bit surprised that there doesn't seem to be a solution which triggers automatically when a file-upload is finished.
Did I miss something?
You might need to handle that event (google.storage.object.finalize) by your own means.
For example, that event can trigger a cloud function (Google Cloud Storage Triggers), which can do various things - from triggering a load job, to implmenting a complex data processing (cleaning, validation, merging, etc.) while the data from the file is being loaded to the BigQuery table.
