How do I write to BigQuery using a schema computed during Dataflow execution?

4
votes

I have the following scenario:

  1. Pipeline A looks up table A in BigQuery, does some computation and returns a list of column names.
  2. This list of columns names is used as the BigQuery schema for output of pipeline B.

Can you please let me know what is the best option to achieve this?

Can pipeline A use TextIO to write the list of column names to temporary or staging location files which are then read by the Pipeline executor to define the schema for pipeline B. If this approach looks fine, can you please let me know if there is a Dataflow utility to read files from temporary or staging location or if the GCS API should be used.

2
google-cloud-dataflow

2 Answers

5
votes

You would need to do the following:

  1. Construct Pipeline A to write to some location such as GCS (any durable location which you can reference when constructing pipeline B would work).
  2. Use the BlockingDataflowPipelineRunner to run and wait till Pipeline A is done.
  3. Construct Pipeline B by using the schema information by reading from the location you defined in step 1.
  4. Run Pipeline B.

I would not use the temporary location because we may clean it up before you get around to constructing Pipeline B. The staging location (if different from the temporary location) can be used. I would also advise to use a unique file name so that if Pipeline A runs multiple times, you don't read in stale results with Pipeline B.

This should help you read from and write to GCS: https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/util/GcsUtil.java

You can get an instance of GcsUtil from the PipelineOptions object: https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/main/java/com/google/cloud/dataflow/sdk/options/GcsOptions.java#L43

1
votes

This is possible with the latest version of Apache Beam. See my more general question with self-answer at Writing different values to different BigQuery tables in Apache Beam .