Is there a way to customize Google provided Cloud Dataflow templates?

Question

I have a requirement wherein I need to create a dataflow job that will transfer data from Google PubSub topics to BigQuery tables by filtering messages based on some attribute which will be part of json data. Using the Google provided templates I can use Google provided "PubSub to BigQuery" template, but that doesn't allow filtering of data.

Is there a way to customize the template "PubSub to Bigquery"? If yes, any links or samples on how to do that would really help.

Answer 1

Everything should be published on GitHub - feel free to clone and modify.

Templates:

• https://github.com/GoogleCloudPlatform/DataflowTemplates

Pub/Sub to BQ:

• https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/main/java/com/google/cloud/teleport/templates/PubSubToBigQuery.java

Also note that the template supports JS UDFs that you could use to customize transformations and filtering without the need to modify the template.

For example, this one provides transformations and filtering:

• https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/master/src/test/resources/JavascriptTextTransformerTest/transform.js