schema free solution to BigQuery Load job

Question

Background

I studied and found that bigQuery doesn't accept schemas defined by online tools (which have different formats, even though meaning is same). So, I found that if I want to load data (where no. of columns keeps varying and increasing dynamically) into a table which has a fixed schema.

Thoughts

What i could do as a workaround is:

1. First check if the data being loaded has extra fields.
2. If it has, a schema mismatch will occur, so first you create a temporary table in BQ and load this data into the table using "autodetect" parameter, which gives me a schema (that is in a format,which BQ accepts schema files).
3. Now i can download this schema file and use it,to update my exsisting table in BQ and load it with appropriate data.

Suggestion

Any thoughts on this, if there is a better approach please share.

Answer 1

We are in the process of releasing a new feature that can update the schema of the destination table within a load/query job. With autodetect and the new feature you can directly load the new data to the existing table, and the schema will be updated as part of the load job. Please stay tuned. The current ETA is 2 weeks.