Cassandra reports:"Unable to complete request: one or more nodes were unavailable" when I use CQL:"select * from User"

Question

Cassandra works in cluster model with 3 nodes.When all nodes are "UP", I use cql “select * from User” in cqlsh，Cassandra returns the right result.But after a node is dead,when I use "select" again,no result returns but reports:"Unable to complete request: one or more nodes were unavailable" . I turned to use cassandra-cli command:"get Users", it returns me the right data without any error. any ideas?

Answer 1

I expect that when you are using CQL you are having a request with a Consistency-Level being "ALL". In this case, it will wait for a reply from all the servers (that host a replica of that node) before returning. As one node is down it fail because it cannot contact the down node.

When you are doing it through Cassandra-cli, I expect that the consistency-level is defaulting to either "QUORUM" or "ONE" or "ANY" and so will happily return you data, even if one replica is down.

Answer 2

cqlsh and cli both default to CL.ONE. I suspect the difference is actually that your cqlsh query says "select all the users" while a "get" in the cli is "select exactly one user."

Answer 3

which strategy do your keyspace placement? if you use NetworkTopologyStrategy, you can try like below:

 CREATE KEYSPACE gxsim with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy' and strategy_options = {'DC1' : 1 , DC2' : 1} AND durable_writes = true;

Answer 4

If your cassandra cluster is in aws, it is then change the configuration in "cassandra.yaml" just change endpoint_snitch to Ec2Snitch again one thing that culd be poossible is your datacenter is actually the "region" of ec2 instance and it should be like "us-east','us-west'. In your case it should be 'eu-west' only.

As per datastax says about it EC2Snitch¶

Use the EC2Snitch for simple cluster deployments on Amazon EC2 where all nodes in the cluster are within a single region. The region is treated as the data center and the availability zones are treated as racks within the data center. For example, if a node is in us-east-1a, us-east is the data center name and 1a is the rack location. Because private IPs are used, this snitch does not work across multiple regions.

When defining your keyspace strategy_options, use the EC2 region name (for example,us-east) as your data center name.

link - http://www.datastax.com/docs/1.0/cluster_architecture/replication http://www.datastax.com/documentation/cql/3.1/cql/cql_using/update_ks_rf_t.html

Answer 5

1. What was the replication factor that you used for the keyspace?
2. How many rows of data does the "users" column family have?

I found myself in a similar situation (yesterday) with replication factor set to 1 and "users" column family having only one row.

Cluster Information: 3 nodes on AWS Same datacenter name

Keyspace name: rf1 SimpleStrategy Replication factor 1 Column Family name: users Querying using cqlsh, default consistency

Scenario 1:

One or two nodes in the cluster were down

I found that the query "select * from users" would return "Unable to complete request: one or more nodes were unavailable" if any of the 3 nodes was down.

Scenario 2:

Node 1 was down. Node 2 was down. Node 3 was up.

The query "select * from users where user_name='abhishek'" would return me the row.

I figured this was the case because the row seemed to be on node 3.

My understanding of the scenario:

When you make the query "select * from users", you are asking Cassandra to return all the rows from the column family. It would not be able to do so since one or more nodes are down and it cannot give you the whole column family since there might be some rows on the nodes that were down.

But the query with the where clause would return the row because it was available on node 3 and node 3 was up.

Does that make sense?

One flaw with this explanation is that I would expect Cassandra to return all the rows that are AVAILABLE in the cluster with "select * from users"

I am going to do some more digging now and will update if I find anything useful.