Verify the data has reached sent to GCP Pub/Sub

Question

We have a project which receives data from sensors and then we send this data to GCP. For this we have used GCP's Pub/Sub model. Issue here is when we pull the messages, they are not in ordered manner. So we are not able to verify that the data we have sent to GCP has reached there or not. Also GCP has mentioned that they don't guarantee the order of messages https://cloud.google.com/pubsub/docs/ordering

Any better way to verify this messages, other than the solutions recommended by GCP.

Answer 1

Ordering is not guaranteed in general in Pub/Sub, it is true. However, when using ordering keys as described in the ordering documentation to which you link, ordering is guaranteed. You would need to set an ordering key on published messages and enable message ordering on your subscription. Right now, the documentation only shows how to do this in Java, though other language examples will be coming soon.

Without using ordering, you could potentially monitor the backlog to see when num_undelivered_messages is 0. However, this has some drawbacks:

1. You would have to continuously query the metric to see its value.
2. The delay in computing the metric is O(minutes) and so it may be stale, resulting in either not tracking messages that were very recently published (resulting in it showing a value less than the actual size of the backlog) or not recording the fact that some messages were delivered and acked (resulting in it showing a value greater than the actual size of the backlog).

In general, it is preferred with Pub/Sub that your subscribers are always running and ready to receive data when it is published. Cloud Pub/Sub guarantees that messages successfully published will be received by subscribers, assuming subscribers are able to receive the messages within the message retention duration, which defaults to seven days.