Need to troubleshoot Bigtable read latency

Question

We have a GKE containerised Internet facing application serving the incoming traffic/request to Bigtable via a load balancer (ssl termination at Nginx). Requests are synchronous in nature. We are using C++ Bigtable client library to interact with Bigtable. See- https://github.com/googleapis/google-cloud-cpp

Most of the "GET" calls are taking 40-50ms which seems to be quite unusual. I have investigated the Bigtable "Key Visualizer Tool" as well and don't see any issue with Rowkey and schema.

In order to confirm that Bigtable is working fine from the Pod, I ran a sample 'hello world' and tried to fetch the response for the sample Rowkeys synchronously and most of the "GET" calls are taking 4-5ms. Now, I'm quite unsure that what is making the "GET" calls over same Rowkeys take so long to return the response if called via application.

Network latency was something to consider but then it would delay the response time for the sample Python 'Hello World" code as well. Also, Bigtable cluster and GKE nodes are in same zone.

Any idea what areas should I be looking into for troubleshooting the same?

Answer 1

The one thing that comes to mind is the initial overhead to create a bigtable::Table object. Are you caching the object between requests? Would it be possible to do so?

A second area would be authentication, though I assume you are using the same authentication method for both the Python 'Hello World' and the C++ application?

If this is not helpful, reach out to me, my email address should be easy to discover from the GitHub site.