Paginate through BigQuery query results using python

1
votes

I see that the BigQuery REST API allows for pagination through results, and I see that the BigQuery python client allows pagination when listing rows in a table (among other things), but I do not see a way to paginate through query results.

The Job makes a call to client.list_rows, but does not give the caller the option of passing in max_results:

https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/bigquery/google/cloud/bigquery/job.py#L2404

It also does not expose sufficient information, as far as I can tell, to be able to build the Table definition (I don't see where to get the query schema) external to the Job and make the list_rows call myself.

Hopefully I'm just missing something...

Help would be greatly appreciated,

--Ben

1
google-bigquery
Running a query creates a temporary anonymous table. You can use the table-based APIs to list its contents, right? – Elliott Brossard
That is certainly what the current job.result call is doing. However it uses private variables _query_results is used to get the schema passed to the Table init function. – Ben
Have you checked the documentation indicating how to page through the results of a query? It includes some code indicating what to do for different kinds of results you may want to have. Is this what you are looking for? – Rodrigo C.
@RodrigoC. You will notice that the API provides a way to page through all kinds of metadata query results (list of datasets, list of tables), and it provides a way to page through a complete dump of a table, but it does not expose a way to page through results of a query. The iterator itself will do the paging using a default buffer size - but there is no way to send in the number of rows or a size at which paging should occur, nor to manually page through results. – Ben

1 Answers