Alter Keyspace on cassandra 3.11 production cluster to switch to NetworkTopologyStrategy

Question

I have a cassandra 3.11 production cluster with 15 nodes. Each node has ~500GB total with replication factor 3. Unfortunately the cluster is setup with Replication 'SimpleStrategy'. I am switching it to 'NetworkTopologyStrategy'. I am looking to understand the caveats of doing so on a production cluster. What should I expect?

Aaron Aaron · Accepted Answer · 2018-06-22T17:05:28

Switching from mSimpleStrategy to NetworkTopologyStrategy in a single data center configuration is very simple. The only caveat of which I would warn, is to make sure you spell the data center name correctly. Failure to do so will cause operations to fail.

One way to ensure that you use the right data center, is to query it from system.local.

cassdba@cqlsh> SELECT data_center FROM system.local;

 data_center
-------------
 west_dc

(1 rows)

Then adjust your keyspace to replicate to that DC:

ALTER KEYSPACE stackoverflow WITH replication = {'class': 'NetworkTopologyStrategy',
    'west_dc': '3'};

Now for multiple data centers, you'll want to make sure that you specify your new data center names correctly, AND that you run a repair (on all nodes) when you're done. This is because SimpleStrategy treats all nodes as a single data center, regardless of their actual DC definition. So you could have 2 replicas in one DC, and only 1 in another.

I have changed RFs for keyspaces on-the-fly several times. Usually, there are no issues. But it's a good idea to run nodetool describecluster when you're done, just to make sure all nodes have schema agreement.

Pro-tip: For future googlers, there is NO BENEFIT to creating keyspaces using SimpleStrategy. All it does, is put you in a position where you have to fix it later. In fact, I would argue that SimpleStrategy should NEVER BE USED.

so when will the data movement commence? In my case since I have specific rack ids now, so I expect my replicas to switch nodes upon this alter keyspace action.

This alone will not cause any adjustments of token range responsibility. If you already have a RF of 3 and so does your new DC definition, you won't need to run a repair, so nothing will stream.

I have a 15 nodes cluster which is divided into 5 racks. So each rack has 3 nodes belonging to it. Since I previously have replication factor 3 and SimpleStrategy, more than 1 replica could have belonged to the same rack. Whereas NetworkStrategy guarantees that no two replicas will belong to the same rack. So shouldn't this cause data to move?

In that case, if you run a repair your secondary or ternary replicas may find a new home. But your primaries will stay the same.

So are you saying that nothing changes until I run a repair?

Correct.

Alter Keyspace on cassandra 3.11 production cluster to switch to NetworkTopologyStrategy

1 Answers