Performance issues with Amazon ELB and CouchDB

Question

I have a CouchDB cluster (replication) consisting of two same boxes. It's master-master cluster which has two documents A and B and A is four times smaller than B, 4KB vs 16KB. In addition there is a HTTP ELB in front of the CouchDB instances which has sticky sessions enabled. I'm observing the following behaviour:

a) When I retrieve document directly from CouchDB, omitting ELB, both document perform well, average ~1000 r/s

b) When I connect through ELB, I'm observing performance degradation for document A ~250 r/s. Document B still performing well ~1000r/s

c) When instead of ELB I setup HAProxy there is no performance degradation.

I did a couple of adjustments in ELB configuration trying different setting in terms of Cross-Zone Load Balancing, Multi AZ, Sticky Sessions but without any improvement.

I'm running CouchDB 1.6.0, and tests are performed by siege 3.0.5.

The only difference between HAProxy and ELB is that ELB is using Connection: keep-alive.

Answer 1

It turned out that CouchDB configuration was the problem. According to CouchDB documentation (http://wiki.apache.org/couchdb/Performance#Network):

The HTTP server library Mochiweb, by default sets the TCP socket option SO_NODELAY to false. This means that small data sent to the TCP socket, like the reply to a document write request (or reading a very small document), will not be sent immediately to the network - TCP will buffer it for a while hoping that it will be asked to send more data through the same socket and then send all the data at once for increased performance.

You can disable that behaviors adding the following to the CouchDB configuration:

[httpd]
socket_options = [{nodelay, true}]

That was the reason why I observed better performance in case when persistent connections were not used.