1
votes

I'm streaming data into a bigquery table, for some reason last few times I checked the data isn't available instantly. The streamed data eventually is available to query, sometimes it looks like it's more than an hour out of sync. But data does eventually appear after some delays.

Is this expected? If it is, what are the ways around it? The table is partitioned by date and we are not "load"-ing data into any particular partition, we are streaming data only for the current day.

Update:

When I say data is not available, I mean running following query doesn't seem to match date/time the data was pushed to Bigquery.

select   max(dateTime) from [nproj.reporting] ;
Post your project name maybe an engineer can check it.Pentium10
When you say "data is not available" - what does it mean ? If the table is partitioned, the _PARTITIONTIME will be NULL while data is still inside streaming system, so make sure you include _PARTITIONTIME IS NULL in your WHERE clause if you do filtering.Mosha Pasumansky
@MoshaPasumansky updated above, the query doesn't seem to show the most recent dateTime rather it looks delayed. This used to work couple of weeks back. I have definitely disabled caching in UI.opensourcegeek
If you can share the job id of a recent job that didn't return the latest data, we can take a look.Pavan Edara
@PavanEdara - Job ID oandmpoc:bquijob_2c54d34_15712bded50 Start Time Sep 10, 2016, 8:17:01 AM End Time Sep 10, 2016, 8:17:02 AMopensourcegeek