Add datacenter with one node to backup existing one

1
votes

I already have a working datacenter with 3 nodes (replication factor 2). I want to add another datacenter with only one node to have all backup data from existing datacenter. The final solution:

dc1: 3 nodes (2 rf)
dc2: 1 node (1 rf)

My application would then connect only to dc1 nodes and send data. If dc1 breaks down I can recover data from dc2 which is on the other physical machine in different location. I could also use dc2 for AI queries or some other task. I'm a newbie in case of cassandra configuration so I want to know if I'm not making some kind of a mistake in my thinking. I'm planing on using this configuration docs to add new dc: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/operations/opsAddDCToCluster.html Is there anything more I should keep in mind to get this to work or some easier solution to have data backup?

Update: It won't only be a backup, we wont to use this second DC for connecting application also when dc1 would be unavailable (ex. power outage).

Update: dc2 is running, I had some problems with coping data from one dc to other and nodetool status didn't show 2 dc's but after fixing firewall rules for port 7000 I managed to connect both dc's and share data between them.

1
cassandra
You will need to consider your consistency level - if you are currently using QUORUM, adding the additional node will start using DC2, you would need to be using LOCAL_QUORUM to ensure DC2 was left out. - Andrew
I think we use default ONE consistency level in our application (default). If I understand correctly when application is contacting seed point (we only contact DC1) it will gather whole claster node information and than it will chose where to connect so I should change it to LOCAL_ONE so it wont connect to new DC2 node. - user1307657
Do you know what are the repercussions of not setting it to LOCAL when adding another dc? I can accept that not all data will be saved when new dc is getting up, but I'd like to be sure if it wont affect already saved data or the once send after dc2 will be fully operable. - user1307657
Setting it to Local_One will prevent it from accessing the 2nd DC in the event that nodes are down in DC1. - Andrew

1 Answers

1
votes

with this approach, your single node will get 2 times more traffic than other nodes. Also, it may add a load to the nodes in dc1 because they will need to collect hints, etc. when node in dc2 is not available. If you need just backup, setup something like medusa, and store data in the cheap environment, like, S3 - but of course, it will require time to restore if you lose the whole DC.

But in reality, you need to think about your high-availability strategy - what will happen with your clients if you lose the primary DC? Is it critical to wait until recovery, or you're really requiring full fault tolerance? I recommend to read the Designing Fault-Tolerant Applications with DataStax and Apache Cassandra™ whitepaper from DataStax - it explains the details of designing really fault tolerant applications.