How to add node to Cassandra cluster which was previously removed?

1
votes

We've been having troubles adding nodes to a Cassandra cluster, and my team is wondering if anyone can spot anything we may be doing wrong, or share their best practices. These nodes were previously members of the cluster, but were removed, and now we are trying to add them back again. The nodes seem to have been removed appropriately: nodetool removenode completed successfully, and nodetool status on the existing nodes showed only the expected nodes, and none of the new nodes. Cluster was fine for a week or two before attempting to add the nodes again.

Before adding these nodes to the cluster again, all the data directories were wiped clean. Then we added the first new node, with autobootstrap, and it appeared to join the cluster just fine, moving past the joining state to normal. So far so good... "describe cluster" seemed to show everything was consistent, and we were about to do a cleanup on the existing nodes and then repair, before adding more nodes.

But then things started to go wrong. Somehow this new node seemed to know about the other previously removed nodes, showing them as down and unreachable (which was true, but how did it now about them?). Read requests to the new node were frequently failing, mostly because data was not found. To avert disaster, the new node was decomissioned and eventually assassinated. And my cluster (and applications) were happy again.

Are there any pre- or post- steps for adding (or removing nodes) we may have missed? I've followed the advice of many other questions posted regarding adding and removing Cassandra nodes. Should I do a complete osreload on the machine before I try adding it as a node again?

I'm grateful for your advice. Again, many of the other questions here has helped me a great deal, but not quite this particular scenario.

2
cassandra
I have a similar problem. If you resolve, a posted answer would be helpful. This is a very poorly documented area of cassandra - Jake
Can you post the system log message during this odd behavior? - dilsingi
Hey Henry, was the node you removed a seed node, and if so, did you remove it's ip from all the cassandra.yaml files seed list? - Christophe Schmitz
What is your configuration in cassandra.yaml, especially the snitch... - Alex Ott
Are you sure you actually deleted all the data including data from system keyspaces? - Simon Fontana Oscarsson

2 Answers

0
votes

I believe to debug this issue , Do a checklist of the following :

  • you need to make sure that you cleaned you commit logs, as when the node was restarted, it might have replayed the commitlogs.

For issue :

Somehow this new node seemed to know about the other previously removed nodes, showing them as down and unreachable (which was true, but how did it now about them?)

  • Make sure the cassandra.yaml config is not old, the issue described as it knows about the previous node means that the Yaml was not updated with the current node list of the cluster.
0
votes

After some wrangling, my team has finally and successfully added new nodes to our Cassandra cluster. I want to share what we learned here (and not leave people hanging).

The biggest problem was the replicator factor on the system_auth table. After adding a new node, our applicant client requests (presumably those connecting to the new node) would fail with errors like 'User userid1 has no SELECT permission on <table xyz.data> or any of its parents'. The answer was specifically inserting our users and roles into the system_auth table on the new node, a using replicator factor increased to cover all nodes. It seems odd that Cassandra does not immediately replicate the system_auth table when adding a new node.

So... to add a node which previously had been added and removed, this is how we did it:

  1. Ensure the node has been previously decommissioned properly with nodetool drain nodetool decommission and nodetool assassinate just to be sure.
  2. Repair the cluster
  3. Perform a clean os install / os reload on the node to be added
  4. Run nodetool status and nodetool describecluster on every node in the cluster to ensure consistent results
  5. Add the node (being sure to autobootstrap)
  6. Repair the system_auth table
  7. Manually insert our users / roles into the system_auth table on the new nodes, using a replicator factor which covered all nodes including the new on
  8. Run nodetool cleanup on the existing nodes
  9. Repair the entire cluster (just for good measure)

We also had some self-inflicted issues with our deployment code, which caused our nodes to be unable to gossip with other nodes. It involved problems accessing TLS truststores, because we encrypt traffic between each nodes. Obviously, if nodes cannot communicate, it's really bad.

My thanks to everyone who offered help, and I hope our experience will help others.