Google Cloud Dataflow table query returns more data than table has

Question

One of the first steps in my Dataflow pipeline involves reading data from a table stored on bigquery. I use the Apache Beam Python SDK 2.3.0 and define this step like:

p | "Read Table" >> beam.io.Read(beam.io.BigQuerySource(query=query))

My query simply reads some columns from the table with no filtering or processing of values at all.

My table holds about 275 GB of data total, however my pipeline reports this step outputting closer to 650 GB of data. Why is the amount of data blowing up so much?

Answer 1

Each element read in contains column names as strings, which increases the size of each row.

In java there is a solution, to load directly into a POJO, without column names.

Unfortunately, this is not available in python. However, please make sure you are not passing around the same elements returned in the PCollection retrieved from the source. You may wish to move the elements into a python object without string names, which should reduce the size. This won't address the size of the first Pcollection created in your pipeline, but can help with downstream PCollections.