how to rebalance cassandra cluster after adding new node

16
votes

I had a 3 node cassandra cluster with replication factor of 2. The nodes were running either dsc1.2.3 or dsc1.2.4. Each node had num_token value of 256 and initial_token was commented. This 3 node cluster was perfectly balanced i.e. each owned around 30% of the data.

One of the nodes crashed so I started a new node and nodetool removed the node that had crashed. The new node got added to the cluster but the two older nodes have most of the data now (47.0% and 52.3%) and the new node has just 0.7% of the data.

The output of nodetool status is

Datacenter: xx-xxxx
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address           Load       Tokens  Owns   Host ID                               Rack
UN  10.xxx.xxx.xxx    649.78 MB  256     47.0%  ba3534b3-3d9f-4db7-844d-39a8f98618f1  1c
UN  10.xxx.xxx.xxx    643.11 MB  256     52.3%  562f7c3f-986a-4ba6-bfda-22a10e384960  1a
UN  10.xxx.xxx.xxx    6.84 MB    256     0.7%   5ba6aff7-79d2-4d62-b5b0-c5c67f1e1791  1c

How do i balance this cluster?

2
cassandra
Can you include the output of 'nodetool status'? - Richard
Did you start the new node with num_tokens = 256 too? - Richard
Edited the post to added output of 'nodetool status' - user2341900
Are you sure you didn't start the new node with num_tokens = 1, then set num_tokens = 256, then restart it? That could explain what you are seeing. - Richard
I can't quite explain what you have seen, but it is important to set these properties before starting Cassandra for the first time. I suggest you decommission the new node, completely wipe its data and commit log directories, then bootstrap it back in. - Richard

2 Answers

6
votes

You didn't mention running a repair on the new node, if indeed you haven't yet done that it's likely the cause of your lack of data on the new node.

Until you run a nodetool repair the new node will only hold the new data that gets written to it or the data that read-repair pulls in. With vnodes you generally shouldn't need to re-balance, if I'm understanding vnodes correctly, but I haven't personally yet moved to using vnodes so I may be wrong about that.

4
votes

It looks like your new node hasn't bootstrapped. Did you add auto_bootstrap=true to your cassandra.yaml?

If you don't want to bootstrap, you can run nodetool repair on the new node and then nodetool cleanup on the two others until the distribution is fair.