Getting “Query too large” in BigQuery

7
votes

I am storing event data in BigQuery, partitioned by day - one table per day. The following query failed:

select count(distinct event) 
from TABLE_DATE_RANGE(my_dataset.my_dataset_events_, SEC_TO_TIMESTAMP(1391212800), SEC_TO_TIMESTAMP(1393631999))

Each table is about 8GB in size.

Has anyone else experienced this error? Seems like it's limited by table size, because in this query, I've only limited it to just one column. When I use a smaller time range, it works.. but the whole point of using BigQuery was its support for large datasets.

1
google-bigquery
hey moderators, please vote to re-open question, since it's not a duplicate question (different error) - Felipe Hoffa
Cool, looking forward to the great feedback. - Pentium10
anyway try: select count(event) from TABLE_DATE_RANGE(my_dataset.my_dataset_events_, SEC_TO_TIMESTAMP(1391212800), SEC_TO_TIMESTAMP(1393631999) group each by event - encc
Question has been re-opened. I added an answer below. Sorry for letting this go to the duplicate bin! - Felipe Hoffa

1 Answers

5
votes

"Query too large" in this case means that the TABLE_RANGE is getting expanded internally to too many tables, generating an internal query that is too large to be processed.

This has 2 workarounds:

  • Query less tables (could you aggregate these tables into a bigger one?).
  • Wait until the BQ team solves this issue internally. Instead of using a workaround, you should be able to run this query unchanged. Just not today :).