Dataflow - Store avro objects to unknown GCS folders

0
votes

I'm have a DataFlow pipeline which consumes JSON messages from Cloud Pub/Sub, converts them to Avro and stores them to Google Cloud Storage.

Is there a way to have DataFlow store each Avro object to a different folder in GCS according to a certain JSON field, if its value can be actually anything?

The goal for this folder structure is to then use PubSub notifications with Cloud Functions to load each resulted file to a different BigQuery table.

1
google-bigquerygoogle-cloud-storagegoogle-cloud-dataflowavrogoogle-cloud-pubsub
What kind of JSON field do you mean? – aga
For instance, a field called: "type" with a string value. – Forepick

1 Answers

1
votes

Does the writeDynamic() function here satisfy what you need? Seems that "Example: Writing CSV files to different directories and with different headers" on that page is similar to what you want to do.