BigQuery streaming insert using template tables data availability issue

3
votes

We have been using BigQuery for over a year now with no issues. We load data as batch jobs every few hours and it usually is instantly available.

We just started experimenting with streaming inserts using template tables. With our first test, we saw no errors and the data showed up instantly. The test created approximately 120 tables. A simple select count (using the web ui) on the tables came up with the right total number of ~8000 rows. After a couple of hours of more streaming, the total dropped to ~1400 rows.

Unsure about what happened, we dropped the dataset, recreated the template table and re-ran the streaming. This time around, the tables showed up right away but the data did not. On our third attempt the tables themselves did not show up for more than a couple of hours. We are on the fourth attempt and this time we only streamed data belonging to one table. The table showed up right away, but it has been over an hour and the data does not show up.

The streaming service uses the latest Java library, inserts only one record at a time and logs the response. The response, without an exception is always {"kind":"bigquery#tableDataInsertAllResponse"} and no errors.

Any help trying to understand what is happening would be great. Thanks.

1
streaminggoogle-bigquery
If you can give us your project/template table/suffixes, we can take a look. That said, there may be caching interactions coming into play... If you delete tables, you may need to wait a few minutes for our caches to clear before you try to stream again. Otherwise we'll have bound your inserts to your deleted table... and then drop those rows (because we think the table they're associated with is gone). - Sean Chen
Sean, Thank you for your quick response. We did wait for around 10 minutes after deleting tables, before we started streaming again. We will try this test once again by deleting the tables, waiting for an hour (just to make sure) and try streaming. If we still see the issue, we will get back to you with the project and other details. Once again, Thank you. - Rajiv Sunkara
Not sure if this is related to the updates that were pushed yesterday and had some serious issues, thus there is a incident on BQ: status.cloud.google.com/incident/bigquery/18013 - Pentium10
@Pentium10 Thank you for the pointer. I don't get any errors when streaming data, so this is probably something else - Rajiv Sunkara
@SeanChen We gave it a few hours of cooling after we deleted the tables. Then we restarted the streaming. Our logs show that we streamed 11094 rows, while we only see 8207 rows in BigQuery. - Rajiv Sunkara

1 Answers

5
votes

Looks like we've identified the issue. It appears there's a race in the template-tables path only that causes our system to think the first chunk of data was deleted by user action (table truncation -- which it obviously wasn't), and is dropped. We've identified the fix and will attempt to push out a fix shortly.

Thanks for letting us know!