Update BigTable row in Apache Beam (Scio)

Question

I have the following use case:

There is a PubSub topic with data I want to aggregate using Scio and then save those aggregates into BigTable.

In my pipeline there is a CountByKey aggregation. What I would like to do is to be able to increment value in BigTable for a given key, preferably using ReadModifyWrite. In the scio-examples there are only updates related to setting column values but there are none of using atomic increment.

I uderstand that I need to create Mutation in order to perform any operation on BigTable, like this:

Mutations.newSetCell(
      FAMILY_NAME, COLUMN_QUALIFIER, ByteString.copyFromUtf8(value.toString), 0L)

How to create UPDATE mutation from Scio / Apache Beam transform to atomically update row in BigTable?

Answer 1

I don't see anything like ReadModifyWrite under com.google.bigtable.v2.Mutation API so not sure if this is possible. I see 2 possible workarounds:

• save events as a Pubsub topic, consume it from a backend service and increment there
• use a raw Bigtable client in a custom DoFn, see BigtableDoFn for inspiration.