Cassandra : Cannot replace address with a node that is already bootstrapped

3
votes

One of the existing nodes in a 5 node Cassandra (3.9) cluster fails to come up.

I noticed the node to be down and tried to restart using the command

service cassandra restart

But the node fails to come and I see the below exception in system.log

ERROR [main] 2017-04-14 10:03:49,959 CassandraDaemon.java:747 - Exception encountered during startup java.lang.RuntimeException: Cannot replace address with a node that is already bootstrapped at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:752) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:648) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:548) ~[apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:385) [apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:601) [apache-cassandra-3.9.jar:3.9] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:730) [apache-cassandra-3.9.jar:3.9] WARN [StorageServiceShutdownHook] 2017-04-14 10:03:49,963 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown WARN [StorageServiceShutdownHook] 2017-04-14 10:51:49,539 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown

Thanks

1
cassandra
The exception message is different. I guess the version of cassandra are different. And I am not trying to replace a dead node. i am just trying bring up an existing node. - Vinod Jayachandran
Have you had a look at you cassandra-env.sh it looks like there is an replace address configured there? - Marko Švaljek
You are right. Somebody from Dev-ops had created an entry with this IP in cassandra-env.sh. After i removed it, the node came up successfully. You saved my day. Thanks a bunch. - Vinod Jayachandran
:) you also made my day ... I like helping ;) - Marko Švaljek

1 Answers

2
votes

Have a look at this guide, basically you have a dead node in the cluster, happens all the time ;)

https://blog.alteroot.org/articles/2014-03-12/replace-a-dead-node-in-cassandra.html

Plus some additional descriptions: https://issues.apache.org/jira/browse/CASSANDRA-7356

Plus check that you also remove the address from: /etc/cassandra/cassandra-env.sh