Cassandra: - Newly added node not visible in nodetool status

Question

I have a single node Cassandra cluster and I added a new node to that cluster. I made sure that the cluster name is same as well as the rack-dc properties are same. I also made sure that the nodes can communicate through all necessary ports. I can cqlsh from both nodes to each other as well as telnet all required ports 7000,7199,9042

cassandra.yaml for new node

1) seed: - "<old_node_ip>,<new_node_ip>"
2) auto_bootstrap: - true
3) listen_address: <new_node_ip>
4) rpc_address: <new_node_ip>

nodetool status command shows only the host node(itself) on both nodes

Running cassandra v 3.11.4 on both with open jdk 8

Am I missing anything?

Update

Here's the log:

java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO  [StorageServiceShutdownHook] 2019-10-21 13:08:14,218 HintsService.java:209 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2019-10-21 13:08:14,219 Gossiper.java:1559 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2019-10-21 13:08:14,219 MessagingService.java:981 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/10.128.1.3] 2019-10-21 13:08:14,220 MessagingService.java:1336 - MessagingService has terminated the accept() thread
INFO  [StorageServiceShutdownHook] 2019-10-21 13:08:14,764 HintsService.java:209 - Paused hints dispatch

it says unable to gossip with peers but I can telnet on JMX port and the seed values are mentioned correct as well

Does the cluster_name match on both nodes? Also, what does the log say about handshaking with the first node? — Aaron
As @Aaron suggested, the log will tell you exactly what is wrong. — Jim Wright
Hmm. Are you using node-to-node SSL? Because that'll be on 7001, and will require matching truststores. Or it could be a case of internal vs. external IP addresses. Either way, your nodes can't gossip with each other, so there's something not quite right along the way. — Aaron
Actually there is no SSL for internode communication, I haven't enabled it. One thing is on my old_node the cassandra is using localhost (127.0.0.1) as shown in nodetool status. Does it has to do with anything with this error ? — Dark Angel

Aaron Aaron · Accepted Answer · 2019-10-21T15:15:06

There are three main things required for getting a new node to join/gossip with an existing node:

cluster_name - The cluster names of the two nodes must match exactly (case-sensitive).
seeds - The new node must use the existing node as a seed node. If you're working in a cloud environment, the "external" (aka "floating") IP addresses must be used.
node-to-node SSL (if used) - The nodes must have matching truststores to connect.

One thing is on my old_node the cassandra is using localhost (127.0.0.1) as shown in nodetool status.

2a. Neither of the nodes can use the home IP of 127.0.0.1.

Cassandra: - Newly added node not visible in nodetool status

Update

2 Answers