BigQuery Datawarehouseing / live data / quotas

0
votes

I'm currently working on a data warehousing project with BigQuery.

BQ used to have this quota:

Daily destination table update limit — 1,000 updates per table per day

While this quota is still in the documentation, I understand that his has been removed according to this blog post:

https://cloud.google.com/blog/products/data-analytics/dml-without-limits-now-in-bigquery

In our project we need live data for which requires a lot of updates. Before this blog post I would have gathered the records e.g. on GCS and pushed them every ~14 minutes into BQ.

With the removal of the table update limit, we could now stream all data immediately into BQ which would be actually vital for our solution as live data are required.

Question: Would you recommend now to stream data directly to BQ? Any objections?

I'm asking this as I think just because the quota has been removed, this doesn't automatically become a best practice. How do you handle the requirement for live data? Another option before has been external data sources with the known limitations.

Thank you for your answers!

1
google-bigquery
Now, worth mention, that DML comes with no limits, thus you can load data to a temp table followed by INSERT AS SELECT statement with no limits, which makes 1K daily quota per table irrelevant. – Adrian

1 Answers

1
votes

This quota never applied to streaming. The quota mentioned in the blog applied to updates via DML queries only - SQL statements with INSERT, UPDATE, MERGE, DELETE statements.

Streaming inserts (via tabledata.InsertAll API, not SQL command) have different limits:

  • Maximum rows per second: 1,000,000
  • Maximum bytes per second: 1 GB

If you do need live data - definitely go with streaming. Note that it is costlier than GCS updates, but if you need fresh data - this is the way to go.