Data missing from Cassandra after adding nodes to the cluster

2
votes

We had a cluster of 6 nodes with replication of 3. 4 new nodes were added just by starting them up and letting them join the cluster, then restarting all nodes to update the seed list. So every node has all other seeds updated and replication factor is still 3. Nodetool status shows all nodes to be UN (new nodes went to UN status fast) and describecluster shows all nodes joined the same schema. Nodetool status also shows old nodes have a lot of data and new nodes very little, just the new data coming in I guess. During the update we did not add auto_bootstrap param to cassandra.yaml, as far as I know default value is true. Token ranges are redistributed so each node owns about 10% of the range. Sorry but not able to copy paste/ screenshot this.

After adding the nodes everything seemed fine, then I noticed some data missing from 3 column families. Maybe there could be more missing, but couldn't spot it at the moment.

End goal is to migrate all the data from 6 old nodes do 4 new nodes and decommission the old nodes.

Questions:

  1. If auto_bootstrap value is true, shouldn't the data be streamed from other nodes to new nodes? Will node be in UJ status until that is completed? My new nodes went to UN pretty fast, doesn't seem like the data is being streamed to them.

  2. Why is my data missing? Shouldn't queries find the nodes with the data and just get it from there?

  3. Most important, how to get the data back? Must still be somewhere on disk. Closest answer I could find is:

You should perform nodetool rebuild on the new nodes after you add them with auto_bootstrap: false

But this is for the case when auto_bootstrap is false. Will rebuild/ repair help?

  1. What is best way to proceed to get all the data in 4 new nodes and decommission 6 old nodes? My plan was to decommission the old nodes one by one, distributing the data that way.

Cassandra version: 2.0.17 Using astyanax latest version, think it's 3.90

1
cassandracassandra-2.0

1 Answers

2
votes

In the future, please try to ask only one question per question.

Are the new nodes also in the seed list? Seed nodes do not stream data, which is why the recommendation of "don't make all nodes seed nodes" exists.

Shouldn't queries find the nodes with the data and just get it from there?

Queries will be directed to the nodes which are responsible for the (token range of) data, not necessarily which node actually may have the data.

how to get the data back?

Try running a nodetool repair on each node. If you have nodes set up in a separate, logical data center, running a nodetool rebuild is actually faster.

Yes, nodetool decommission will readjust the token range assignments, and distribute data. But I'd run the repair operation before doing that, as your original nodes are probably the only ones with the missing data.

The other option, would be do decommission all new nodes, and start over. Then re-add them, making sure they get their data. If they do not, run a repair on that one node. Then add the next, rinse, repeat.

Cassandra version: 2.0.17 Using astyanax latest version, think it's 3.90

Yikes! That version of Cassandra is 5 years old, and Astyanax hasn't had a PR in 2 years. There have been many improvements and bug fixes to Cassandra during that time, so I would highly recommend upgrading and getting off of anything dependent on Thrift.