What happens when a replica node goes down, with consistency level of quorum?

1
votes

Scenario:

  • Total Nodes: 3 [ A, B, C ]
  • Replication factor: 2
  • Write consistency: Quorum (2 replicas need to ack)
  • Read consistency: Quorum

  • Nodes partion ranges:

    • A [ primary 1-25, replica 26-50 ]
    • B [ primary 26-50, replica 51-75 ]
    • C [ primary 51-75, replica 1-25 ]

Question:

Suppose I need to insert data 30 and node A is down. What would be the behavior of Cassandra in this situation? Would Cassandra be able to write the data and report success back to the driver (even though the replica node is down and Cassandra needs 2 nodes to acknowledge a write)?

3
cassandradatastaxdatastax-enterprisecassandra-3.0

3 Answers

3
votes

You only have 1 replica available for the write (B), so you'll get error on write (UnavailableException).

It's better to design your consistency levels / replication factor so that you can tolerate node's failure for a token range (consider bumping your RF to 3).

Also better not to try to solve the availability by following the eventual consistency path (R + W <= N), e.g. putting W=1 in this case. We've tried that and operationally it was not worth the effort.

2
votes

Is there strong reason behind RF=2? given the scenario, Quorum will not be satisfied in a node down scenario and your writes will fail. I suggest you to revisit your RF.

1
votes

You have identified one of the key reasons why RF=2 is not an advised replication factor for highly available Cassandra deployments. What will happen is depending on driver behavior (tokenaware on or off).

  1. Node B or C will be chosen as the coordinator
  2. The coordinator will attempt to write to both B and A because a Quorum of 2 is 2
  3. The coordinator will note that node A has not acknowledged the write and thus report back to the client that a Quorum was unable to be achieved.

Note, this does not mean that the write to Node B failed... in fact the value is written to Node B and the coordinator will store a hint for Node A. However you have not achieved your consistency goal so it is likely advisable that you attempt the write again until the node comes back up in most situations. In this specific situation, you are doing effectively ALL which is not going to give the expected behavior in node failure situations.

TLDR, don't use Quorum with RF=2