BigQuery streaming data not available instantly

Question

Since couple of days some data i am streaming to bigquery is not available instantly (as it normally happens) within bigquery web ui after being inserted successfully.

My use case consists of inserting thousand of lines using :

bigquery.tabledata().insertAll(...)

The results of the streaming inserts into the table are : (i am also checking for insertErrors to be sure as described here):

BigQuery insert status : {"kind":"bigquery#tableDataInsertAllResponse"}
BigQuery insert errors  : null

Total number of lines available in bigquery web ui is different that total inserted.

I would be grateful for any help.

Bigquery project details :

Project ID  : favorable-beach-87616
Table : mtp_UA_xxxx_1_20150410

Project dependencies on google libraries:

compile 'com.google.api-client:google-api-client:1.19.0'
compile 'com.google.http-client:google-http-client:1.19.0'
compile 'com.google.http-client:google-http-client-jackson2:1.19.0'
compile 'com.google.oauth-client:google-oauth-client:1.19.0'
compile 'com.google.oauth-client:google-oauth-client-servlet:1.19.0'
compile 'com.google.apis:google-api-services-bigquery:v2-rev171-1.19.0'
compile 'com.google.api-client:google-api-client:1.17.0-rc'

Great thanks in advance for your help!

Jordan Tigani Jordan Tigani · Accepted Answer · 2015-04-10T18:06:12

When you say the total number of lines available in the web UI, do you mean the number of rows that show up in the 'details' pane on the table, or the number of rows that are returned if you do a SELECT COUNT(*) query?

If the former, that is expected, since that counter only returns the number of rows that have been flushed to long-term storage (as opposed to the short-term storage buffers the streaming data originally gets written to). This is admittedly confusing, and we are working on a fix.

If the latter, the rows don't show up in a query, that is more concerning. If that is the case, please let us know and we'll investigate.

BigQuery streaming data not available instantly

1 Answers