Cassandra Cluster Set up - Unable to gossip with any seeds

Question

I am trying to set up a 3 node Cassandra VM cluster.

I installed cassandra from datastax package on individual vms and then modified the following:

Seed - vm1 (set the ip address in all the vm configs)
Updated the config with listen_address as the host ip, added the rpc_broadcast_address
Added the cassandra ports in the firewall rules to allow for inter vm communication
Also tried connecting to the vms using SSH

After trying all of this, I started the cassandra seed node, it comes up fine and when I check for the status using nodetool, I see that the node status is UN (Up/Normal). I then start cassandra on the other cluster nodes. It fails to start up with unable to gossip with any seeds error. Any help will be greatly appreciated!

Stack Trace:

java.lang.RuntimeException: Unable to gossip with any seeds
    at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1334) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:540) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:783) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:720) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) ~[apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:537) [apache-cassandra-2.1.9.jar:2.1.9]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:626) [apache-cassandra-2.1.9.jar:2.1.9]

Thanks!

I added the following lines in the /etc/sysconfig/iptables file and also flushed/restarted iptables service -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m tcp --dport 7000 -j ACCEPT -A INPUT -p tcp -m tcp --dport 7199 -j ACCEPT -A INPUT -p tcp -m tcp --dport 9160 -j ACCEPT — user3421442

user3421442 user3421442 · Accepted Answer · 2015-09-24T21:14:53

Ok, after spending a day on this, found the issue. It was indeed a firewall issue. In addition to entering the rules in iptables, ensure that the ACCEPT rules appear before the REJECT rules -> ordering matters. After changing the order and restarting the iptables service and cassandra, it started working, the nodes were able to gossip with the seed.

Cassandra Cluster Set up - Unable to gossip with any seeds

1 Answers