Google BigQuery - Streaming Data Into BigQuery

Question

I'am using Google BigQuery for my project Right now i'am trying to insert a new row to BQ based on this https://cloud.google.com/bigquery/streaming-data-into-bigquery#bigquery-stream-data-java

private void insertRowsToBQ(MyCustomObject data) {
    String datasetName = "mydatasetname";
    String tableName = "mytablename";
    Map<String, Object> rowContent = new HashMap<>();
    rowContent.put("field_1", data.getdata1());
    rowContent.put("field_2", data.getdata2());
    rowContent.put("field_3", data.getdata3());
    rowContent.put("field_4", data.getdata4());

    try {
        BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
        TableId tableId = TableId.of(datasetName, tableName);
        InsertAllResponse response =
                bigquery.insertAll(
                        InsertAllRequest.newBuilder(tableId)
                                .addRow(rowContent)
                                .build());

        if (response.hasErrors()) {
            for (Map.Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()) {
                Logger.error("Response error: \n" + entry.getValue());
            }
        }
        Logger.info("Rows successfully inserted into table");
    } catch (BigQueryException e) {
        Logger.error("Insert operation not performed \n" + e.toString());
    }
}

the code is working fine, got no error log BUT when i'am trying to view it on google console https://console.cloud.google.com/bigquery?project=myprojectname

select * from `myprojectname.mydatasetname.mytablename` where DATE(_PARTITIONTIME) = "2021-03-24"

the data on google BQ console, the data is not displayed. It's turn out that the data is delayed more than 1 hour, until it can be view on the BQ

is this expected? or there are some issue?

i've been trying recreate the dataset and table, but still no luck

You should not view the data when you click on the preview button of your table. But if you perform a query, without any partition value, you should vue your data. — guillaume blaquiere

Sunandini Ravada Sunandini Ravada · Accepted Answer · 2021-05-10T08:46:46

As per documentation on data availability,looks like when streaming to a partitioned table, data in the streaming buffer has a NULL value for the _PARTITIONTIME pseudo column. Also i could see we have some fields through which we can check the streamingBuffer.oldestEntryTime etc.,

For a table where we are doing streaming insert its also mentioned that Data can take up to 90 minutes to become available for copy operations.

https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataavailability

Google BigQuery - Streaming Data Into BigQuery

1 Answers