BigQuery, querying the streaming buffer & a partition

0
votes

I would like to query data from BigQuery which is in the most recent partition, but also in the streaming buffer.

Ideally I would like to stream data into a more granular partition than a day, maybe using a decorator:

mydataset.table$2017030112

Following this, I would query the table for the partition & the buffer.

SELECT * FROM `mydataset.table` 
WHERE _PARTITIONTIME IS NULL 
AND _PARTITIONTIME IS 2017030112

The result should return just the partition data, regardless if it is in the buffer or not.

The issues / questions:

  • Is it possible to partition more granular than by DAY?
  • Would a query accessing the buffer and a partition charge just for the partition?
  • Is there a better approach?
1
google-bigquery

1 Answers

0
votes

Google Cloud Support here!

  • It's not possible to partition more granular than a day.

  • You are charged for the bytes that your query scans, When you query the streaming buffer, moreover, you are querying what's called an unpartitioned partition.

  • Not that I can think of. This is the way to go.

I hope that helps