Manage load and timeout on couchbase

Question

I have a couchbase cluster with 4 hosts (4 CPU, 32 GB ram).

This cluster is used by thousands processes which get data from it. I have 8K permanent connections on the cluster (data from couchbase ui).

The bandwidth uses by couchbase is 50MB/s per hosts to send data (don't seem very heavy).

But I get client side timeout. About 4 per seconds and I don't understand why.

I set somaxconn and tcp_syn_backlog to 2048 per hosts but it doesn't change something.

Do you have any idea ?

Edit from comments:

I have this kind of logs from my client:

Error from Couchbase, passing in non-blocking mode (EXC: Key=u'fieldmetadata', RC=0x17[Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout], Operational Error, Results=1, C Source=(src/multiresult.c,316), Tracing Output={"fieldmetadata": {"i": 15890195098182236653, "s": "kv:Unknown", "b": "MetaDataV3", "r": "ltg-aus-couchbase-3:11210", "t": 2500000}}>)

I have 3 nodes in my cluster. All clients and cluster is on Azure VM.

What language and which version of Couchbase Server and the SDK are you using? Can you post debug logs from the application? — Perry krug
FWIW, if it's Java...a) upgrade the SDk and b) check for garbage collection pauses. Those are the two most common solutions to timeouts. Not saying there aren't others, just easy to rule those out. — Perry krug
It's used with python. Couchbase 5.5 server and couchbase==2.5.8 for python — Kiva
Okay, we'll still need a bit more information to help debug...do you have logs from the client? Also make sure you're re-using the same client object as much as possible across multiple threads rather than constantly creating and destroying it. Which python framework are you using? Also, 8k connections isn't indicative of a problem by itself, but does that match with how many application instances you have? — Perry krug
Timeouts can be either client, network or server...a quick way of ruling it out is to take a packet capture and look for the request that times out. How long does it take to get to the server, how long does it take to respond, etc. If it goes to the server and then doesn't come back for a long time, we can investigate the server performance...if it goes to the server and comes back quickly, we can investigate the SDK/application. — Perry krug

Jon Strabala Jon Strabala · Accepted Answer · 2019-11-28T17:12:19

Kiva, as per a similar thread on the couchbase forums

Can you set LCB_LOGLEVEL=5 in your environment (i.e. as an environment variable) and re-run the code? ... which may help reveal more information as to why it’s timing out.

Manage load and timeout on couchbase

1 Answers