Set custom widow duration for Pub/Sub to Text Files on Cloud Storage dataflow from GCP console

Question

I am trying to implement Pub/Sub to Text Files on Cloud Storage dataflow template from GCP console with custom window duration like 1hr instead of 5min(which is default).

Any suggestions would be of great help. Thank you.

Answer 1

Through the GUI, it seems impossible. However, if you use directly the template on github you have a parameters for this.

The rollout of the new versions could be done soon, but in case of emergency, you can use directly the Java code.

Answer 2

Since it was already mentioned, when using Dataflow Google-provided templates it is not possible to change the pipeline parameters that are different to introduced runtime ones, you might consider for the particular Pub/Sub to Cloud Storage Text template either:

• Clone the template source, integrating it in your own local Beam project and adjusting the customization for the required default parameters:

   @Default.String("1h")
   String getWindowDuration();
   void setWindowDuration(String value)
  

or

• Cloning the source pipeline code and predefine the runtime parameter for window duration in the ValueProvider interface:

  ValueProvider<String> getWindowDuration(); void
  setWindowDuration(ValueProvider<String> value);