Cassandra multi dc replication nodetool rebuild issue and schema mismatch

Question

Team, I have a Cassandra cluster of 6 nodes and I've added a new datacenter to the existing setup of 5 nodes. I’ve followed all the steps but getting the below error when I run nodetool rebuild on the new dc’s nodes.

nodetool rebuild -- datacenter1

nodetool: Unable to find sufficient sources for streaming range (389537640310009811,390720100939923035] in keyspace system_distributed
See 'nodetool help' or 'nodetool help <command>'.

Nodetool status

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address    Load   Tokens   Owns (effective) Host ID               Rack
UN 172.21.201.205 1.75 GiB 256     16.5%      ff0accd4-c33a-4984-967f-3ec763fe5414 rack1
UN 172.21.201.45 1.55 GiB 256     17.0%      d3ac5afa-d561-43ee-89e2-db1d20c59b38 rack1
UN 172.21.201.44 2.37 GiB 256     17.0%      73d8e6c6-0aa3-4a91-80fc-8c7068c78a64 rack1
UN 172.21.201.207 1020.15 MiB 256     16.0%      5751ea7d-b339-43b3-bcfe-89fcbc60dea0 rack1
UN 172.21.201.46 1.64 GiB 256     17.0%      1c1afbfc-6a4b-40f0-a4c3-1eaa543eb2d5 rack1
UN 172.21.201.206 1.13 GiB 256     17.2%      b11bfef9-e708-45cc-9ab3-e52983834096 rack1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address    Load   Tokens   Owns (effective) Host ID               Rack
UN 10.41.6.155  983.91 KiB 256     17.3%      bf7244bb-70dc-4d91-8131-cbe4886f09e7 rack1
UN 10.41.6.157  946.36 KiB 256     15.5%      5499e7cc-db23-4163-8f0c-8f437f61bd6f rack1
UN 10.41.6.156  1.14 MiB 256     15.3%      f27e94a6-7e1c-4177-9f88-36d821a7808d rack1
UN 10.41.6.159  659.3 KiB 256     17.3%      453e97df-5b83-4798-9e5e-a13bbb33acee rack1
UN 10.41.6.158  909.51 KiB 256     18.2%      a4bc046a-e2ef-4fd4-9ab7-18be642a4d5a rack1
UN 10.41.6.160  1.08 MiB 256     15.5%      267cf9d0-cd55-4186-a737-998443125b19 rack1

node tool describe cluster status

    #nodetool describecluster
    Cluster Information:
       Name: Recommendation
       Snitch: org.apache.cassandra.locator.GossipingPropertyFileSnitch
       DynamicEndPointSnitch: enabled
       Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
       Schema versions:
         ea63e099-37c5-3d7b-9ace-32f4c833653d: [10.41.6.155, 10.41.6.157, 10.41.6.156, 10.41.6.159, 10.41.6.158, 10.41.6.160]     
         fd507b64-3070-3ffd-8217-f45f2f188dfc: [172.21.201.205, 172.21.201.45, 172.21.201.44, 172.21.201.207, 172.21.201.206, 172.21.201.46]

OLD DC Cassandra version - 3.1.1
NEW DC Cassandra version - 3.11.4

Can someone quickly help me fix this issue?

utkarsh-devops utkarsh-devops · Accepted Answer · 2020-11-11T20:01:15

Changing the replication factor of the system_distributed keyspace to NetworkTopologyStrategy to include both dcs worked for me.

Command executed

ALTER KEYSPACE system_distributed WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'datacenter1' : 2, 'dc1' : 2};

Output

system_distributed | True | {'class': 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '2', 'dc1': '2'}

Cassandra multi dc replication nodetool rebuild issue and schema mismatch

2 Answers