Limit of Big Query data size

Question

I have large SQL select to do on regular basis such as :

  select * from table1 where ...

which 50 millions records, from Python client. I could not find in Google Compute doc, the limit of exporting large amount of data from Big Query through SQL from Python client.

Exporting is only limited to 1GO per file....

https://cloud.google.com/bigquery/docs/exporting-data#bigquery-export-table-gcs-python

Is there any reference/experience on very large SQL select done on remote client and the limitation of Big Query ?

EDIT Question 2 :

Why we cannot receive more than 128Mo of data when using Order by clause ?

Your question is unclear . . . are you trying to move 50 million rows from BQ to Python? — Gordon Linoff

Mikhail Berlyant Mikhail Berlyant · Accepted Answer · 2017-04-23T16:05:27

Is there any reference/experience on very large SQL select?

From Queries Quota Policy:

Maximum response size: 128 MB compressed - Sizes vary depending on  
compression ratios for the data; the actual response size may be  
significantly larger than 128 MB.

At the same time - size is unlimited when returning large query results:

If you plan to run a query that might return larger results, you can
set allowLargeResults to true in your job configuration. Configuring
large results requires you to specify a destination table. You incur
storage charges for the destination table.

Update to address comment:

This answer points you to documentation/reference you asked for.
It is relevant for any client that you use to retrieve data or result of query.

In your case - if size of your query is too big to retrieve because of above mentioned limitations - I would recommend to store your result in destination table and than use tabledata.list API to retrieve data stored in that destination table

Limit of Big Query data size

1 Answers