Uneven data distribution on cassandra nodes

1
votes

enter image description hereI had a cassandra cluster of 4 nodes in aws singapore region and I have to migrate it to mumbai region so I created a new datacenter in mumbai region added it to the singapore cluster and run rebuild command on all the new nodes one by one. Data on previous nodes = 150GB/node approx. Data on new nodes = 200GB/node on two node, 100GB on one node.

I am not able to figure out why this is happening it should be 200GB/node. I am using cassandra 3.9 and using vnodes on new datacenter and single token on old datacenter.

I want to figure out what is the reason for this.

I tried to rebuild it finished in few seconds and RF is 3 nodetool status is showing all node UP and normal

1
cassandranodetool
Can you please provide values for " Owns (effective),Tokens,Load" columns in nodetool? - nevsv
i added a nodetool snapshot - deenbandhu
Does both of your cluster running same version of Cassandra? - nevsv
yes both are using 3.9 - deenbandhu

1 Answers

1
votes

Try to rebuild last node once again, it can be due to unfinished rebuild process, and review the logs to see any errors.

Also, can you please provide output of nodetool status? And, what are replication factor in each dc?

EDIT:

1) Try to run repair on 100gb node.

2) Stop the problematic node, remove all the data from data folder and rebuild once again (it is possible that you will need to start the node with node replace options).