7
votes

It looks like keyspace replication is not happening correctly in my Cassandra setup, I need some ideas in troubleshooting this. I have configured multi datacenter cluster but to begin with I have set the keyspace to use SimpleStrategy with RF 3.

The column families exist:

cqlsh:kairosdb> select columnfamily_name from system.schema_columnfamilies where keyspace_name = 'kairosdb';

 columnfamily_name
-------------------
       data_points
     row_key_index
      string_index

(3 rows)

but I am unable to query on them:

cqlsh:kairosdb> select count(*) from data_points limit 100000;
 Traceback (most recent call last):
  File "/usr/bin/cqlsh", line 957, in perform_simple_statement
    rows = self.session.execute(statement, trace=self.tracing_enabled)
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.1.post.zip/cassandra-driver-2.1.1.post/cassandra/cluster.py", line 1282, in execute
    result = future.result(timeout)
  File "/usr/share/cassandra/lib/cassandra-driver-internal-only-2.1.1.post.zip/cassandra-driver-2.1.1.post/cassandra/cluster.py", line 2776, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 1}

This is how I setup the multi datacenter setup:

  1. Use 2 nodes from each DC in the seeds list
  2. Use org.apache.cassandra.locator.GossipingPropertyFileSnitch for the Snitch
  3. Specify different name for DC in the cassandra-rackdc.properties as appropriate for the node

Here's the keyspace is created:

cqlsh:kairosdb> describe keyspace kairosdb;

CREATE KEYSPACE kairosdb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

Any ideas what I can do to troubleshoot this?

2
Can you try level one local? Did you have data in the keyspace and alter it to RF 3? - phact
What is level one local? - vrtx54234
In multi DC you can specify local vs each to specify if you want to wait for responses from a remote data center. datastax.com/documentation/cassandra/2.0/cassandra/dml/… - phact
Can you share details about the process you took to set up multi-dc and configure your RF 3 - phact
I edited the original post describing how I setup the multi-dc cluster and how the keyspace is created. Is SimpleStrategy for replication compatible with multi-dc setup? Since this was not working, I removed two rings, moved back to SimpleSnitch and this time the keyspace got created correctly. so I am wondering if the keyspace creation is incorrect. - vrtx54234

2 Answers

3
votes

In your keyspace creation you have this syntax:

CREATE KEYSPACE kairosdb WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

However if you wish to use replication across data centers (DC) then you need to use NetworkTopologyStrategy, so for example:

CREATE KEYSPACE kairosdb WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': 3, 'DC2': 3};

As per the following documentation link, "Use NetworkTopologyStrategy when you have (or plan to have) your cluster deployed across multiple data centers..."

http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureDataDistributeReplication_c.html

-3
votes

First run below command on any of your nodes:

describe keyspace system_auth;

Check the class of this keyspace,whether it's SimpleStrategy or NetworkTopologyStrategy.

It should be NetworkTopologyStrategy if you have configured Cassandra for multiple DCs.

If the output returns the class as SimpleStrategy, you need to ALTER this keyspace.

COMMAND:

ALTER KEYSPACE "system_auth" WITH REPLICATION = 'class' : 'NetworkTopologyStrategy', 'dc1' : 3, 'dc2' : 2};

After having done this,try creating a new keyspace to check. You will stop getting the Consistency error that you have been facing.