Cloud Dataflow: Using Google-provided PubSub to BigQuery template with multiple json entries

Question

I am using the Google-provided template for PubSub to BigQuery with no customizations. I am trying to put multiple entries(rows) into a single json payload onto the queue and then have the DataFlow template insert all entries(rows) into the BigQuery table. I have tried providing a newline delimited json payload like is required when loading data into BigQuery via the console. However, I am only able to get the first entry to insert into the table.

Does the default DataFlow template only take a single entry(row)?

Answer 1

Currently the Google-provided template only accepts a single JSON record as payload within the Cloud Pub/Sub message and will not detect any newline delimited JSON. Look for this to change in the near future as additional supported formats are added to the template.