cassandra timeout during write when nodetool repair is going on

Question

I have cassandra running on 3 nodes, where consistency is 1. Nodetool repair is also scheduled on server.

Now I am observing that during nodetool repair was happening, my write is timed out, even in retries which finally bringing down my data ingestion service.

com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency LOCAL_ONE (1 replica were required but only 0 acknowledged the write) at com.datastax.driver.core.exceptions.WriteTimeoutException.copy(WriteTimeoutException.java:73)

Can we do write operation while nodetool repair is going on?

Answer 1

Usually this happen when the node is overloaded - nodetool repair requires intensive input/output when comparing the data on different servers, and this add an additional load to server, together with additional garbage collection, etc., and could lead to the timeouts. Plus, repair may trigger compaction that will also add an additional IO load.

Please check the system.log on your Cassandra boxes, and maybe tune some parameters - Java garbage collection, Java heap size, etc.