Cassandra: What happens when an entire datacenter goes down?

0
votes

I have a Cassandra cluster with 3 datacenters of 6 nodes each - DC1, DC2 and DC3 like this. Lets say I have a NetworkTopologyStrategy and while creating a keyspace I do,

CREATE KEYSPACE test WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'DC1' : 3 ,'DC2' : 3,'DC3' : 3 }

which makes sure that any write is replicated on 3 nodes in each datacenter only after which the write is successful.

What happens if an entire datacenter goes down ?

Lets say all nodes in Dc3 goes down! In this case, a write request to update a table, even with Write consistency level of ONE (Consistency Level: A write must be written to the commit log and memtable of at least one replica node. I would assume it needs to be written to atleast one node in each datacenter.) would fail right ? This request will try to update atleast 3 nodes in DC3, which it cant because of all nodes being down.

https://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html

1
cassandradatastax

1 Answers

1
votes

If you are writing with CONSISTENCY LEVEL of ONE your writes will not fail.

Explanation:

Your keyspace specifies 3 replicas in each datacenter which ensures that there will be 9 copies of each row of your data. When you write at CONSISTENCY LEVEL of ONE the co-ordinator will give success once the data is written to memtable and commitlog of a single node (only a single node, not one node per datacenter).

So what happens about replicas? The data is stored as hinted hintoff (default 3 hours) which gets replicated once nodes are up (in your case data center 3 is up). If the nodes are not up before 3 hours then it will try to repair the data using builtin repair mechanism .

which makes sure that any write is replicated on 3 nodes in each datacenter only after which the write is successful

Your this statement does not stand true if you are writing with CONSISTENCY LEVEL ONE.

For more details on Multiple Datacenter Write.

Three replicas in each data center: This configuration tolerates either the failure of a one node per replication group at a strong consistency level of LOCAL_QUORUM or multiple node failures per data center using consistency level ONE.