Cloud pubsub slow poll rate

Question

I have a pubsub topic, with one subscription, and two different subscribers are pulling from it. Using stackdriver, I can see that the subscription has ~1000 messages. Each subscriber runs the following poll loop:

client = pubsub.Client()
topic = client.topic(topic_name)
subscription = pubsub.Subscription(subscription_name)

while True:
    messages = subscription.pull(return_immediately=True, max_messages=100, client=client)
    print len(messages)
    # put messages in local queue for later processing. Those processes will ack the subsription

My issue is a slow poll rate - even though I have plenty of messages waiting to be polled, I'm getting only several messages each time. Also, lots of responses are back without any messages. According to stackdriver, my messages pulled rate is ~1.5 messages/sec.

I tried to use return_immediately=False, and it improved it a bit - the pull rate increased to ~2.5 messages/sec, but still - not the rate I would expect to have.

Any ideas how to increase pull rate? Any pubsub poll best practices?

Kamal Aboul-Hosn Kamal Aboul-Hosn · Accepted Answer · 2016-08-29T12:32:53

In order to increase your pull rate, you need to have more than one outstanding pull request at a time. How many depends on how fast and from how many places you publish. You'll need at least a few outstanding at all times. As soon as one of them returns, create another pull request. That way, whenever Cloud Pub/Sub is ready to deliver messages to your subscriber, you have requests waiting to receive messages.

Cloud pubsub slow poll rate

1 Answers