Difference Between Partitions and Multiple Outputs in a ParDo?

Question

I'm new to Apache Beam and using the Python SDK. Let's say I have a PCollection with some elements that look like this:

{"item": "foo", "color": "green", "date": "2020-10-30"}
{"item": "bar", "color": "blue", "date": "2020-10-30"}
{"item": "bar", "color": "green", "date": "2020-10-30"}
{"item": "foo", "color": "blue", "date": "2020-10-30"}

If I want to split this into several PCollections based on some element attribute, it seems that I have the choice of Partition or ParDo with tags (and using with_outputs() when the ParDo is invoked).

Are there guidelines when I would use Partition over ParDo? It seems like Partition is meant for splitting a PCollection where the resulting PCollections all have the same schema (link), whereas a ParDo could be used to accomplish that, but is better used for splitting a PCollection into multiple PCollections each with a different schema (link). Am I understanding the documentation correctly?

Haiyun Tian Haiyun Tian · Accepted Answer · 2020-11-02T10:07:50

ParDo specifies a generic parallel processing and the runner will manage this "fanout", while Partition has no intention for a parallel but it is designed for spliting a collection into a sequence of sub-collections with the logic determined by a function which you create.

A typical user case for partition can be that of grouping students by percentile and passing groups to their corresponding downstream steps. Notice that different group of student can have different downstream process, and this is just not what ParDo is designed for.

In addition, another difference between Partition and ParDo is that the former must have a predefined partition number while the latter has no such concept.

Difference Between Partitions and Multiple Outputs in a ParDo?

1 Answers