In Google Dataflow streaming, I want to use record timestamps provided in Pub/Sub message attributes rather than the published timestamp.
PubsubIO.Read<T> pubsubIO = PubsubIO.readProtos(type).fromSubscription(options.getPubSubSubscription())
pubsubIO.withTimestampAttribute("eventTimestamp");
When I added withTimestampAttribute to the code, it worked fine with local direct runner, but failed when using Google Cloud Dataflow Runner.
Workflow failed. Causes: Step setup_resource_/subscriptions/xxxxxx: Set up of resource /subscriptions/xxxxxx failed, Creating watermark tracking pubsub subscription projects/xxxxxx to topic projects/xxxxxx failed with error: User not authorized to perform this action.
The service account to run dataflow have admin roles on dataflow, pubsub, etc, so I assume we can rule out the IAM issue.
Not sure if I missed any required configurations, e.g extra setup when create the pub/sub topic and subscription?
