Cloud Pub/Sub and Cloud Dataflow fix worker to topic

1
votes

I have a Google Cloud Pub/Sub and Cloud Dataflow stream processing architecture, i need guaranteed message ordering. Is it possible to set the subscription message flow so that the dataflow workers are fixed per topic, so the messages are routed through the same dataflow worker and therefore should achieve message ordering.

Thanks

1
google-cloud-platformgoogle-cloud-dataflowgoogle-cloud-pubsub
Pub/Sub doesn't guarantee that it maintains message ordering, so I am doubtful this will accomplish what you want. – Matt Wilbert
if i am unable to guarantee message ordering, what would you suggest as an alternative to cloud pubsub? – Al Hennessey
What I would like to suggest is doing something that doesn't require that guarantee, but if that would require too much of an architectural change, I believe RabbitMQ or Kafka would be options with stronger ordering guarantees. You might look at the series of articles beginning at jack-vanlightly.com/blog/2017/12/4/… – Matt Wilbert
@AlHennessey does your message have a timestamp value? If it does you could do a sort by at the end of every apache-beam streaming window and thus guarantee message ordering (note this is not true for late messages that are processed past their original timing window). – Haris Nadeem
This of course takes computational resources and may not even be desirable. – Haris Nadeem

1 Answers

0
votes

When you say you need guaranteed message ordering, do you mean: 1. Order in the final result matters in this case you can you could attach a timestamp to every event in the publisher and make the subscriber store the messages in some underlying data store. 2. Order of processed messages matters in this case you should do the same plus have a subscriber check the persistent storage to know the next message it must process.

You can read more on this in this link .