PubSub REST subscription pull not returning with all messages

3
votes

We use the REST service API to pull messages from a PubSub subscription. Messages ready to be serviced are acknowledged, leaving other messages unacknowledged to be serviced during a later execution cycle.

During an execution cycle, we send a single reqeust to the pull service REST API with returnImmediately=true and maxMessages=100.

While testing we encountered a situation when only 3 "old" messages would be returned during each execution cycle. Newly published messages were never included in the request to pull. We verified new messages were successfully arriving at the subscription by monitoring the Undelivered messages in Stackdriver monitoring.

  • Does the pull REST API not include all undelivered messages?
  • Does it ignore the maxMessages parameter?
  • How should all messages, up to the maximum specified, be read with the REST API?

Notes:

We worked around the problem by sending 2 parallel requests to the pull API and merging the results. We found the workaround (requiring parallel requests) discussed here.

Update Feb. 22, 2018

I wrote an article on our blog that explains why we forced to use the PubSub service REST API.

1
node.jsgoogle-cloud-pubsub

1 Answers

3
votes

A single pull call will not necessarily return all undelivered messages, especially when returnImmediately is set to true. The pull promises to return at most maxMessages, it does not mean that it will always return maxMessages if there are that many messages available.

The pull API tries to balance returning more messages with keeping end-to-end latency low. It would rather return a few messages quickly than wait a long time to return more messages. Messages need to be retrieved from storage or from other servers and so sometimes all of those messages aren't immediately available to be delivered. A subsequent pull request would then receive other messages that were retrieved later.

If you want to maximize the chance of receiving more messages with a pull request, set returnImmediately to false. This still will not guarantee that messages will all be delivered in a single pull request, even if maxMessages is greater than the number of messages yet to be delivered. You should still send subsequent pull requests (or even more ideally, several pull requests at the same time) to retrieve all of the messages.

Alternatively, you should consider switching to the Google Cloud Pub/Sub client libraries, which handle all of this under the hood and deliver messages to a callback you specify as soon as they are available.