Cassandra - avoid nodetool cleanup

6
votes

If we have added new nodes to a C* ring, do we need to run "nodetool cleanup" to get rid of the data that has now been assigned elsewhere? Or is this going to happen anyway during normal compactions? During normal compactions, does C* remove data that does no longer belong on this node, or do we need to run "nodetoool cleanup" for that? Asking because "cleanup" takes forever and crashes the node before finishing.

If we need to run "nodetool cleanup", is there a way to find out which nodes now have data they should no longer own? (i.e data that now belongs on the new nodes, but is still present on the old nodes because no one removed it. This is the data that "nodetool cleanup" would remove.) We have RF=3 and two data centers, each of which has a complete copy of the data. I assume we need to run cleanup on all nodes in the data center where we have added nodes, because each row on the new node used to be on another node (primary), plus two copies (replicas) on two other nodes.

1
cassandracassandra-3.0nodetool
nodetool cleanup just caused a 25+ million row loss on my setup. As per the documentation, I waited until new nodes were up and running (UN) but the documentation does not mention the schemas need to be sync'ed. Turns out nodetool describecluster showed a problem where the schemas were out of sync, but the nodes were up normal UN. So without knowing this I ran nodetool cleanup. 30 minutes later it had finished destroying the work of 20+ days. - Jose Fonseca

1 Answers

5
votes

If you are on Apache Cassandra 1.2 or newer, cleanup checks the meta data on files so that it only does something if it needs to. So you are safe to just run it on every node, and only those nodes with extra data will do something. The data will not be removed during the normal compaction process, you have to call cleanup to remove it.