google pub-sub setMaxMessages

Question

I'm using google pubsub to fetch messages synchronously

    com.google.pubsub.v1.PullRequest.Builder pullRequestBuilder = PullRequest.newBuilder().setSubscription(subscriptionName).setReturnImmediately(returnImmediately);
    if (maxMessages != 0) {
        pullRequestBuilder.setMaxMessages(maxMessages);
    }

    PullRequest pullRequest = pullRequestBuilder.build();
    PullResponse pullResponse = (PullResponse)this.subscriber.pullCallable().call(pullRequest);
    return pullResponse.getReceivedMessagesList();

I saw in the documentation:

setMaxMessages
public PullRequest setMaxMessages(java.lang.Integer maxMessages)
The maximum number of messages returned for this request. The Pub/Sub system may return fewer than the number specified.
Parameters:
maxMessages - maxMessages or null for none

In my code I pass MAX_INT to avoid any max messages limitation

but I see my java code fetches messages one by one.

What is the right way to skip the max limit?

How can I know it's not just a pub-sub bug?

I sometimes get 0 messages even though there are some in the pubsub.

Kamal Aboul-Hosn Kamal Aboul-Hosn · Accepted Answer · 2018-05-09T12:18:20

When you are using the synchronous pull method, setting the maxMessages field does not guarantee you will receive that many messages, even if there are more than that number available. There is a tradeoff between end-to-end latency of messages and number of messages returned in a single response. If the service waits until maxMessages are available to be sent, then the messages that are buffered and waiting will have a higher end-to-end latency.

The maximum number of messages that can be returned in a single response is 1,000, regardless of how high maxMessages is set above that. If you want to make it more likely that your response will contain more messages, make sure you set returnImmediately to false. If this field is true, then the service attempts to return as quickly as it possibly can, meaning it may not wait for messages to be loaded before returning. This is likely why you see 0 messages being returned in some responses.

If you are trying to maximize throughput, you'll want to use StreamingPull, or even better, the Cloud Pub/Sub client libraries, which use StreamingPull under the hood. That way, messages can be deliver messages to an open streaming connection as soon as they are available.

google pub-sub setMaxMessages

1 Answers