Google Pub Sub Message Pull latency Issue with .Net Client

Question

I am pulling messages continuously from Google PubSub. All going good, except the time taken to fetch the message is around 12 to 15 seconds and its not acceptable in our case. Following is my CallTiming settings:

 public CallSettings GetPullSetting()
    {
        CallTiming timing = CallTiming.FromRetry(new RetrySettings(
                retryBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 0, 50), new TimeSpan(0, 0, 5), 1),
                timeoutBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 18, 0), new TimeSpan(0, 0, 20), 1),
                totalExpiration: Google.Api.Gax.Expiration.FromTimeout(TimeSpan.FromMilliseconds(600000))));

        return CallSettings.FromCallTiming(timing);
    }

I trying all kind of combination to reduce this latency to max 3 seconds.

One observation is that whenever a message is pulled successfully, and in the very next iteration of the pull if there is a message on pubsub, it fetches that message immediately. That means if message is found in consecutive pull latency is very low.

But the problem is, say in one iteration I get Deadline exceeded exception since pubsub has no message. Then I push a message in pubsub for the next iteration. At this point it takes lot of time (13 to 16 seconds). So the condition to reproduce this issue is I shall have one failed attempt to pull the message.

Code pasted here:

public void PullTest()
        {
            var cont = true;
            SubscriberSettings settings = new SubscriberSettings()
            {
                PullSettings = GetPullSetting()
            };
            SubscriberClient subscriberClient = SubscriberClient.Create(settings: settings);
            var subscriberName = new SubscriptionName("project-name", "subscription-name");
            while (cont)
            {
                try
                {
                    PullResponse response = subscriberClient.Pull(subscriberName, returnImmediately: false, maxMessages: 1);
                    System.Diagnostics.Trace.WriteLine(">>>>>> " + DateTime.Now.ToString());
                    System.Diagnostics.Trace.WriteLine(">>>>>> " + "Job Recieved" + response.ReceivedMessages.ToList().FirstOrDefault());
                    subscriberClient.Acknowledge(subscriberName, new List<string>() { response.ReceivedMessages.ToList().FirstOrDefault().AckId });
                }
                catch (Exception ex)
                {
                    System.Diagnostics.Trace.WriteLine(">>>>>> " + DateTime.Now.ToString());
                    System.Diagnostics.Trace.WriteLine(">>>>>> " + ex.Message);
                }
            }
        }

 public CallSettings GetPullSetting()
        {
            CallTiming timing = CallTiming.FromRetry(new RetrySettings(
                    retryBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 0, 50), new TimeSpan(0, 0, 5), 1),
                    timeoutBackoff: new BackoffSettings(new TimeSpan(0, 0, 0, 18, 0), new TimeSpan(0, 0, 20), 1),
                    totalExpiration: Google.Api.Gax.Expiration.FromTimeout(TimeSpan.FromMilliseconds(600000))));

            return CallSettings.FromCallTiming(timing);
        }

Naman Bairagi Naman Bairagi · Accepted Answer · 2017-02-28T02:03:53

You have a problem with back off between consecutive pulls when timeout deadline exceeds. You are essentially looking for a long polling solution, for which you either need to reduce backoff timings to close to 0, or use a custom connection/client to retry without delay.

Google Pub Sub Message Pull latency Issue with .Net Client

2 Answers