What happens when I change pipeline options while executing on the Dataflow worker?

0
votes

What happens when if I change pipeline options while executing on the Dataflow worker?

The public documentation for PipelineOptions is: https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/options/PipelineOptions

1
google-cloud-dataflow

1 Answers

2
votes

PipelineOptions is safe to access from multiple threads and is global per Dataflow worker. You should only get values from PipelineOptions and not update PipelineOptions or mutate any values stored within PipelineOptions on the Dataflow Worker. This means that any object placed within PipelineOptions must be thread-safe as it may be accessed from many threads at the same time. Note that there are many types within Java such as SimpleDateFormat which may seem to be thread-safe but are not.