Cassandra clustering fail over-High Avialability

0
votes

I have configured a cassandra clustter with 3 nodes

Node1(192.168.0.2) , Node2(192.168.0.3), Node3(192.168.0.4)

https://i.stack.imgur.com/vXqsi.png

Created a keyspace 'test' with replication factor as 2.

Create KEYSPACE test WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 2}

When I stop either Node2 or Node3 (one at a time and both at one time) , I am able to do the CRUD operations on the keyspace.table.

When I stop Node1 and try to update/create a row from Node4 or Node3, getting following error although Node3 and Node4 are up and running-:

All host(s) tried for query failed (tried: /192.168.0.4:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections))) com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /192.168.0.4:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)))

I am not sure how Cassandra elects a leader if a leader node dies.

2
cassandracluster-computingfailover
There is no concept of leader in cassandra... check if you can telnet to host (192.168.0.4) on port 9042 – undefined_variable
Could you provide more information about the Consistency level used on queires (this has a huge impact on behavior you are expecting)? Are you using a driver or accessing using cqlsh? – Arthur Landim
@undefined_variable .... yes I am able to telnet from my local desktop to all the nodes on port 9042. – UAnand
@ArthurLandim .... I am using DBeaver Enterprise and connecting to the nodes by cassandra cql to execute my queries. – UAnand
@ArthurLandim.... The queries are listed below -: CREATE KEYSPACE test WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 2} CREATE TABLE test.emp( emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint, emp_phone varint ) INSERT INTO test.emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES(11,'JOhn', 'Fort Worth', 434333333, 150000) – UAnand

2 Answers

0
votes

So, you are using replication_factor 2, so only 2 nodes will have a replica of you keyspace (not all the 3 nodes).

You can use this calculator to find out if your setup have a decent consistency. In your case the result is You can survive the loss of no nodes without impacting the application

0
votes

I think I wasn't clear at response. The replication factor is about how many copies of your data will exists. The consistency level is how many copies your client will wait to be made before get an response from server. Ex: All your nodes are up. The client make a CQL with CL Quorum, the server will copy the data in 2 nodes (3/2 + 1) and reply to client, in background it will copy the data at the third node as well.

In your example, if you shutdown 2 nodes of a 3 node cluster you will never achieve an QUORUM to make requests (with CL QUORUM), so you have to use consistency level ONE, once the nodes are up again, cassandra will copy the data on them. One thing that can happen is: before cassandra copy the data on other 2 nodes, the client make a request for node1 or node2 and the data is not there yet.