multi node cluster for cassandra- unable to gossip each other

0
votes

To run 2 instances of cassandra in 2 azure VM's, i have configured seeds, listen address , rpc address in cassandra.yaml file . i have set the different port in cassandra.env.sh file. cluster name is same in both yml files. but i am unable to see the running status of both nodes in nodetool status.

VM1:
cassandra.yaml:
seeds: "10.50.1.71,10.50.1.70" 
start_rpc: true
rpc_address: 10.50.1.70
listen_address: 10.50.1.70

 cassandra-env.sh:
JMS port:7199

VM 2:
cassandra.yaml:
seeds: "10.50.1.71,10.50.1.70" 
start_rpc: true
rpc_address: 10.50.1.71
listen_address: 10.50.1.71

cassandra-env.sh:
JMS port:7299

nodetool status:

Datacenter: datacenter1 ======================= 
Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack 
UN 10.50.1.71 103.66 KiB 256 100.0% c8320c90-3a12-4e11-96b5-ad9a9c69ea11 rack1

system.log

INFO [main] 2020-08-24 13:40:53,988 Gossiper.java:1684 - Waiting for gossip to settle... DEBUG [GossipStage:1] 2020-08-24 13:40:56,420 Gossiper.java:1266 - Shadow request received, adding all states INFO [GossipStage:1] 2020-08-24 13:40:56,426 OutboundTcpConnection.java:108 - OutboundTcpConnection using coalescing strategy DISABLED DEBUG [MessagingService-Outgoing-/10.50.1.70-Gossip] 2020-08-24 13:40:56,434 OutboundTcpConnection.java:425 - Attempting to connect to /10.50.1.70
DEBUG [MessagingService-Outgoing-/10.50.1.70-Gossip] 2020-08-24 13:40:56,439 OutboundTcpConnection.java:546 - Unable to connect to /10.50.1.70 java.net.NoRouteToHostException: No route to host at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_265] at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_265] at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:645) ~[na:1.8.0_265] at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.11.4.jar:3.11.4] at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:434) [apache-cassandra-3.11.4.jar:3.11.4] 
WARN SystemKeyspace.java:1087-No host ID found,created 067b4c10-2191-42a3-a3c7-dc44acf9084c(Note: This should happen exactly once per node) StorageService.java:569-Unable to gossip with any peers but continuing anyway since node is in its own seed list StorageService.java:723-Loading persisted ring state StorageService.java:841-Starting up server gossip
1
cassandra
There isn't enough info in your post. What's the reason you think they can't gossip? Please add that to your post. And what does "unable to see running status" mean? – Erick Ramirez
when i ckecked to see the nodes status using nodetool status , only single node is getting updated as seen below instead of both 2 nodes Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.50.1.71 103.66 KiB 256 100.0% c8320c90-3a12-4e11-96b5-ad9a9c69ea11 rack1 – anusha reddy
That's still insufficient info. Start the second node with just .70 as the seed. If that errors out, it will provide you clues. What warnings/errors are you seeing in the logs? You haven't provided any actionable info that would allow anyone to assist you. – Erick Ramirez
Unless you're using a Thrift-based API (you shouldn't be) you shouldn't need start_rpc: true. – Aaron
Also, the system.log is your best tool for diagnosing gossip issues. Usually it has to do with port 7000 not being open on both sides (or 7001 for SSL). – Aaron

1 Answers

0
votes

It may be caused by starting VM1 then VM2 (Escenario 1) or by launching both nodes at the same time (Escenario 2). The issue will be solved if you start VM2 first, let it complete the bootstrap (wait until it becomes Up and Normal - UN), then start VM1.

Rationale:

  1. While starting the VM1 node, it looks at the seed list; because the first IP is VM2's, it will first try to contact the second node, which may be stopped (Escenario 1) or bootstrapping (Escenario 2).
  2. VM1 evaluates the second IP address in the seed list, which is its own IP address, so it starts as a standalone node
  3. While starting the VM2 node; the first IP address in the seed list its own, so it also starts as a standalone node

If you start VM2 first, wait for it to be UN, then the VM1 you will allow first to have a standalone node, to which all the other nodes will be added.