What should we do differently after the table decorators bug fix?

2
votes

As reported here: BigQuery: BigQuery: Why does Table Range Decorators return wrong result sometimes?

and here https://code.google.com/p/google-bigquery/issues/detail?id=160

The table decorators bug was allegedly fixed. However, my regular process that runs every hour, and query records that were recently added, still returns partial results.

What steps will reproduce the problem? 1. Pick a table with on going batch uploads taht have started within the past 24 hours. make sure that data is still being written to table 2. run the following query: Select * from (Select count() from [MYDATASET.MYTABLE@-]), (Select count() FROM [MYDATASET.MYTABLE])

3.Notice the different results.

My question: Should we do things differently? Is there a time window that table decorators remain incorrect?

For the BQ guys: this query shows the problem: job_kqJ_yZh674kTag4I-ibCPnAMXOw

Many thanks for any help...

1
google-bigquery

1 Answers

1
votes

Copying the answer from https://code.google.com/p/google-bigquery/issues/detail?id=160#c12 here so it doesn't get lost:

That job appears to be running correctly. It just has a timestamp range that eliminates some of the data in the table.

When using timestamp range decorators, the timestamps are compared to a commit time, when data that you loaded is made available as part of the table. In order to make sure you include the commit time, use a range decorator with a lower bound that's definitely before the commit time, such as the time you started your load job, and an upper bound that's definitely after the commit time, such as a time when you observed the loaded data in the table or >= the last_modified_time of the table.