My Dataflow pipeline collates event data into typed per session and per user PCollections output. I have employed GroupByKey for events keyed by session id. Sessions are grouped into parent types keyed by user id and device id using the same pattern at this next level of hierarchy. So a single session might generate many events, but in turn a single user might generate many sessions.
I would now like to summarize this data across each level of the hierarchy. I have used a StateSpec declaration to persist state at the event level. So for example, an event count property can be incremented in my event processing ParDo. (Use Case : generating an error event per session across all users for example.)
But as each ParDo is static - I cannot access the ValueState outside of the ParDo context even though my understanding is this state is maintained at the Window scope. (Maybe this is by design.) Is there a way to access this Window level state using the Beam State persistence lib in another ParDo than where it was originally declared? Like as if I could declare it at the pipeline level?
I understand that this may introduce some performance overhead as the framework must manage concurrency, but the actual processing seems negligible. (Just incrementing values.) So I would prefer to write this to a window level state field rather than percolate values up via my hierarchy.
