Insert older than 7 days data into partitioned BigQuery tables

Question

The documentation attests that it is possible to insert data older than 7 days (and not older than 1 year) into partitioned tables.

But whenever I try streaming one month old data using Go BigQuery Client, it returns an error: "You can only stream to date range within 7 days in the past and 3 days in the future relative to the current date."

How can I stream data older than 7 days using the Go client?

Edit 1: Here is the table schema:

bigquery.TableMetadata{
    Schema: bigquery.Schema{
        {Name: "page_id", Required: true, Type: bigquery.IntegerFieldType},
        {Name: "user_id", Required: false, Type: bigquery.IntegerFieldType},
        {Name: "hit_time", Required: true, Type: bigquery.TimestampFieldType},
    },
    TimePartitioning: &bigquery.TimePartitioning{Field: "hit_time", RequirePartitionFilter: true},
}

Can you show me what your table looks like? I'm not sure what type of partitioning you are using. — Felipe Hoffa
You can still insert data into partitions using batch ! (Since it's old data, maybe you can batch it ?) — vdolez
Yeah, this is an option. But the data still comes as a stream, so I would have to create a batch with one entry if I wanted up to date tables. — Fabrício

Fabrício Fabrício · Accepted Answer · 2018-10-24T19:25:24

According to this issue, this is a new feature that "should be fully rolled out very soon". I got my project whitelisted to try it and I was able to insert older data into the tables.

Insert older than 7 days data into partitioned BigQuery tables

1 Answers