How to Load data into bigquery from buckets partitioned with Year/Month/Day

1
votes

We have a data stored on gcp bucket in below format - gs:/gcptest/Year=2020/Month=06/day=18/test1.parquet and so many files under the day=18 folder. I want to create a table in bigquery with the columns present in the files and partitioned by Year,Month,Day that is present on the file path. So that when I will load the data into table I can just select the path from the gcp bucket and load the data which will partitioned by values of Year/Month/Day present on the path

1
google-bigquery

1 Answers

1
votes

BigQuery supports loading externally partitioned data in Avro, Parquet, ORC, CSV and JSON formats that is stored on Cloud Storage using a default hive partitioning layout.

Support is currently limited to the BigQuery web UI, command-line tool, and REST API.

You can see more in Loading externally partitioned data documentation
Also see how to Query externally partitioned data