Increasing of used space while rebalancing Cassandra cluster

Question

Just an example: I have 2 Cassandra nodes, 1Gb data per each node, replication factor is 1. I use single column family with Leveled compaction with 100Mb sstable size, like this:

create column family ColFamily with key_validation_class=UTF8Type 
  and compaction_strategy=LeveledCompactionStrategy 
  and compaction_strategy_options={sstable_size_in_mb: 100};

I want to add additional node. The data will be rebalanced across 3 nodes: ~0,667 Mb per node. Right?

But how the used space will be increased on each node while the process of rebalancing is being in progress? What will be the peak?

Which Cassandra version are you using? If 1.2, what have you set num_tokens to? — Richard

Richard Richard · Accepted Answer · 2013-04-26T10:20:36

Before Cassandra 1.2 and virtual nodes, you have to do the redistribution of data yourself after adding a new node.

If your two nodes are currently balanced i.e. have 50% of the ring each, then the tokens will be

node1: 0
node2: 85070591730234615865843651857942052864

(or shifted, but I'll assume node1 has token 0). The token for node2 is 2^127/2. You want to end up with

node1: 0
node2: 56713727820156410577229101238628035242
node3: 113427455640312821154458202477256070484

where the token for node2 is 2^127/3, and for node3 is (2^127/3)*2. What you need to do is bootstrap node3 with initial_token set to the token above. This copies data from node1, since node3's token precedes to node1's (the token ring is wrapped around).

Now node3 will have 1/6 of the data, node2 will still have 1/2 and node1 will store 1/2 but only be responsible for 1/3. You could now run 'nodetool cleanup' on node1 to remove the data that it copied over to node3. This will reduce node1's data to approx 677MB.

Now you need to move node2's token to its final place. This copies data from node2 to node3, bringing node3 up to its quota of 1/3 of the data, approx 667 MB. Now you can run 'nodetool cleanup' on node2 to remove the data it has just copied to node3. Now the rebalancing is complete.

This means no node ever stores more than 1 GB of data during the rebalancing.

In general, if you had more nodes or higher replication factor, you can always do the rebalancing without increasing the data stored on any existing nodes if you run cleanup after each move on the node just moved.

Finally, if you had Cassandra 1.2 and virtual nodes, the tokens can be chosen randomly which gives even load as soon as you add a new node, with no need for any rebalancing (manual or automatic). This is not only easier, it saves copying a constant fraction of your data around the cluster just to add one node.

Increasing of used space while rebalancing Cassandra cluster

2 Answers