what if the coordinator is down which has hints of the write request for the dead replca node

Question

I have came across reading about hinted handoff for cassandra write requests. which stores hints about dead replica node in coordinator node. For more info http://www.datastax.com/documentation/cassandra/2.0/cassandra/dml/dml_about_hh_c.html

Then if the dead replica node comes alive coordinator node will take the hint info and replay the write request to that replica node. My question is what if the coordinator node is dead when the replca node comes alive?? Is there any backup coordinator node to replay the hinted writes to replica node if the coordinator is down?

Answer 1

No. Even worse, the hints would stop being accumulated on the coordinator node after a certain amount of time of the target downtime - why wasting space if that node may be dead forever...

So once your node comes back from the downtime, you should run nodetool repair -pr on that node to ensure that data gets replicated to it from another nodes.

Answer 2

Cassandra uses a failover strategy driven by the LoadBalancingPolicy -> "we’ll try the next node in the Iterator provided by this policy"

Source in comment-section

Answer 3

If the coordinator fails mid-request, we're in a similar situation to what we had in the non-distributed case: the client is in the dark and has no choice but to retry. The only difference is that the client can reconnect to any node in the cluster immediately. https://www.datastax.com/dev/blog/how-cassandra-deals-with-replica-failure

Clente will try connect any other node in cluster.