Google Dataflow: create templates with runtime parameters

Question

In Data flow, I need to pass start Date and end date as runtime arguments and query bigquery for that date range and write output to day wise folders.

When we use ValueProvider, getStartDate().get() method is throwing java.lang.RuntimeException: Not called from a runtime context. If I hardcode some value when getStartDate().get().isAccessible() is false, template is being generated but the runtime arguments are not reflecting in job. It is always running with the hardcoded value during template creation.

Any suggestions ?

Answer 1

BigQueryIO takes a ValueProvider of the query. The easiest way to do this is to pass the query text as the runtime value.

NestedValueProvider could help you create the query string from another value provider, alas, NestedValueProvider only support one input ValueProvider at a time. So you could concatenate your start and end dates into a single value and then do the split.