Get Pub/Sub message's messageId field from PubSubIO with Apache Beam

4
votes

Is it possible the get somehow the messageId field of a Pub/Sub message in a DoFn after using the PubSubIO Beam source to read the messages?

I need the default id which was assigned by the Pub/Sub service. I want to log it for debugging purposes.

Using a custom attribute for the unique id and the withIdAttribute() method is not possible for me, because I have no influence on the publisher in this case.

I use the 2.2.0 version of the Dataflow Java SDK.

2
javagoogle-cloud-dataflowapache-beamgoogle-cloud-pubsub
It doesn't look like the Java SDK gives access to the messageId or the publishTime of the message. The PubsubMessage class only has member variables for the message payload and the message attributes. This should be requested as a new feature. – Andrew Nguonly
issues.apache.org/jira/browse/BEAM-3489 – Csaba Kassai
Is there no way to integrate API calls and try to get message data that way? The API call returns the messageID field as well. – Tudormi
While reading the message, have you tried using this API ? PubsubIO.readMessagesWithMessageId() – Balkrishan Aggarwal

2 Answers

1
votes

Support for reading the Pubsub message id was added starting with Beam v2.16.0. To turn it on, replace .readMessages() with .readMessagesWithMessageId() in your pipeline setup then it is as easy as message.getMessageId() after that change.

0
votes

For debugging purposes you can use the seek option.

It creates a snapshot of the messages which you can replay when needed.