Cassandra vnodes: can I lower the number on slower nodes and expect rebalancing to occur automatically?

1
votes

I am running a small Cassandra 2.2.1 test cluster with 3 computers in it. Two of them are i7s and one is a somewhat slower i5, but I didn't bother when first setting things up to give this slower machine a proportionally lower number of vnodes, as I thought things would be IO bound (they all have SSDs and 16GB RAM). They're all on default 256 vnodes.

I'm finding Cassandra actually to be quite CPU intensive though and this i5 seems to be holding things up (running 100%x4 on HTOP). Can I reduce its number of Vnodes to say, 192 in cassandra.yaml, restart just that node, and expect the whole 3-machine cluster to rebalance? Share out the new vnode allocations automatically etc? Or do I have to do something on the i5 alone, or perhaps on all 3 machines with nodetool?

I am finding the documentation somewhat unclear on this as it talks mostly about adding new nodes rather than changing vnodes on existing hardware.

2
cassandra
What is your keyspace(s) definition? – piotrwest
@piotrwest: I'm using SimpleStrategy and replication factor 2. – Thomas Browne
Then it's alright. I was asking because if you had RF=3, all nodes would have entire set of the data, thus changing vnodes wouldn't do much. – piotrwest

2 Answers

4
votes

A simple restart won't allow you to change the number of vnodes.

You'll need to do a nodetool decommission or nodetool removenode for the i5 node, then stop Cassandra on that node, scrub all the data, update num_tokens, then start Cassandra back up and let it rejoin the cluster as an empty new node with the reduced number of tokens. Then Cassandra will rebalance with the reduced number of tokens on the i5 box.

At least that's how it is in 2.1.9, but probably it is the same in the newer branches too.

0
votes

I don't have sufficient reputation to comment, but to answer Thomas Browne's question: Don't do that! If you drop the tables and keyspaces, you'll lose all the data on all the nodes. What you'll want to do is stop cassandra, go to your data directory on the node you're modifying, and rm -rf * the data in that directory. Eg: rm -rf /var/lib/cassandra/*