BigQuery table from Google Cloud Storage

Question

I created a BigQuery table from Google Cloud Storage. I use the option: gs://my_bucket/*.csv

Google Cloud Storage URIs begin with "gs://" and specify the bucket and object you want to load.

Example: gs://mybucket/path/to/mydata.csv

You can use a wildcard to load multiple files, for example gs://mybucket/path/to/my*

My Bucket contains multiple CSV files.

Every time I add a new CSV file, this is not detected by table, meaning if I have in GCS:

• file1.csv

• file2.csv

At the moment I created the table, it will always keep reading those 2 files only. When I add file3.csv and so on, it never updates the table.

Answer 1

When you load data from a bucket, that data is loaded based on the (matching) objects in that bucket at the time of the load.

New objects added are not automatically loaded.

If you want to do that, you have a few options:

• Rely on external tables to query the data in the bucket directly: https://cloud.google.com/bigquery/external-data-sources
• use a Cloud Functions trigger on the bucket to create a load job - https://cloud.google.com/functions/docs/concepts/events-triggers#triggers
• Use Cloud Dataflow and the GCSIO to read files and create load jobs: https://beam.apache.org/documentation/sdks/pydoc/2.0.0/_modules/apache_beam/io/gcp/gcsio.html - Cloud Composer can be used to configure a cron job around this.