Google Cloud Pub/Sub console view messages extremely slow

0
votes

Help? I have an IoT device and am using MQTT to Pub/Sub. I want to view messages in the Pub/Sub console but it can take about an hour to show up, and sometimes they do not show up at all. I click View Messages and it shows messages from days ago. Wait a few minutes and click again, some newer messages show up, but not the one I need to see. Keep doing that for an hour or so and eventually I get to the end, which may or may not have the most current message. If I come back in a few days then I'll see the message I was hoping to see. How can I rapidly see the most recent messages?

Here's a screenshot of my subscription settings. I've tried both Retain = Yes and No.

Subscription settings

Here's screenshots of the first pull I did today. Days old messages. I did have "Enable ACK messages" checked, and per the comment at the top "Only a few messages will be pulled at a time." But as you can see in later screenshots I turned that off with the same effect.

First pull, top of the page

First pull, bottom of the page

And here we go a few minutes later. Not quite seeing today's messages but getting closer.

Second pull

Wait a few minutes, okay good now one more message showed up but Oops! Lost the latest one. Hmm.

Third pull

Let's turn off "Enable ACK messages". Nope, still the same.

Enable ACK messages unchecked, top of the page

Enable ACK messages unchecked, bottom of the page

Wait another five minutes, Oops! Lost that message on the 17th again. Still nothing from today. It's been about twenty minutes and I'm giving up for now. Will check again in an hour or so.

Do you have this issue? How do you work around it?

Second pull with ACK disabled

1
google-cloud-pubsub
I cant tell you about the console settings. But what you can do easily is to use a cloud function with pub/sub. Just keep the default code and print the whole payload. it will be quite easy to see your pub/sub messages in cloud functions log - Hsn
@Hsn that worked beautifully. - Christopher de Vidal

1 Answers

0
votes

If there have been a lot of messages published to a topic, the console is not going to be a good way to look for and view specific messages. The console is a good way to get a sample of the messages that are being published, but there are no guarantees about which messages will be returned. Additionally, Cloud Pub/Sub does not offer any ordering guarantees unless one is using ordering keys and an ordered subscription.

When you retrieve messages via the console, a Pull request is sent to a server. The server will try to fulfill this request as quickly as possible with messages and it will take messages from whichever distributed servers return messages first. That means the messages could come that were published at any point in time. If you click the button to make more requests, this will happen again.

If you are trying to examine specific messages, you should set up a subscriber to read the messages and examine the messages you are looking for, likely by using the Cloud Pub/Sub client libraries.