We have a 3 node Cassandra cluster, and we created a keyspace with a replication factor of 1. After we altered the keyspace's replication factor to 2 (and performed a nodetool repair for that key-space), I see it shares a Merkle tree with all three nodes.
My question, is why would it share with 3 nodes, and not only with 2 nodes?
When triggering a token redistribution, all nodes must be contacted. This is necessary, because all nodes must share an even amount of token range responsibility. To accomplish that, the token range responsibility will change per node; especially with such a small cluster.
Case-in-point, you had a 3 node cluster with a RF of 1. That means each node was responsible for 33.33% of the total tokens (-2^63 to 2^63 -1).
By increasing the RF from 1 to 2, while keeping the number of nodes constant, you are effectively doubling the amount data your cluster will store. Therefore, each node is now responsible for 66.67% of the data.
If you were to further increase your RF to 3, then each node would be responsible for 100% of your data, effectively storing all of your data on each node.
The reason it's talking to all of the other nodes is that some rows will go from node 1 to node 2, others from node 1 to node 3, some from node 2 to node 1, etc. Each row potentially will get redistributed, and where they start/end includes the entire nodes in the data center. Each row get's re-calculated to where it belongs. Does that make sense?
