How to Customize GCP Dataflow template?

Question

I intend to use Pub/Sub to Text Files on Cloud Storage dataflow template with few customizations such as process(massage) the PubSub message before writing to Cloud Storage.

I have apache-beam pipeline code written but confused on how to deploy it. The parameters it consumes will be exactly the same as Pub/Sub to Text Files on Cloud Storage

From documentation I understand that I can use one of the Google-provided templates or create your own. But instead of creating my own template is there a better way to customize Google-provided template as it suffices most of my requirements

Answer 1

I think we are in an all or nothing situation. the only customization that does not need creating your own template is exposed through parameters and they do no accept Ptransforms.

Since you need to modify the ingested Pub/sub messages, you will need to create your own PTransform, integrate it in your pipeline and generate the associated template.

Given that it's only one little addition, your best call is to clone the template sources and copy them in your own local Beam project (or generate it from the cloned project). !!Do not modify the example template itself. Just the code example available here!! Generate the template as stated in the documentation. You are good to call it.