Cassandra Nodes DN - Simple 2 Node Cluster Configuration Fail

Question

I created a 2 node data-center, cluster. This took quite a bit of work because of some confusing and out of date documentation. My problem is now, when I use the nodetool status command the remote machine's state is DN or Down Normal. From my understanding the state should be UN or Up Normal.

Also, what is interesting, what is interesting is that I receive this constant server output:

WARN  17:31:51 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:31:51 Setup task failed with error, rescheduling
WARN  17:32:01 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:01 Setup task failed with error, rescheduling
WARN  17:32:11 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:11 Setup task failed with error, rescheduling
WARN  17:32:21 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:21 Setup task failed with error, rescheduling
WARN  17:32:31 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:31 Setup task failed with error, rescheduling
WARN  17:32:41 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:41 Setup task failed with error, rescheduling
WARN  17:32:51 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:32:51 Setup task failed with error, rescheduling
WARN  17:33:01 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:01 Setup task failed with error, rescheduling
WARN  17:33:11 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:11 Setup task failed with error, rescheduling
WARN  17:33:21 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:21 Setup task failed with error, rescheduling
WARN  17:33:31 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:31 Setup task failed with error, rescheduling
WARN  17:33:41 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:41 Setup task failed with error, rescheduling
WARN  17:33:51 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:33:51 Setup task failed with error, rescheduling
WARN  17:34:01 CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  17:34:01 Setup task failed with error, rescheduling

This output is on the machine that is not the seed. The main seed does not output this constantly.

Also, if it was not clear. If I am running the nodetool status program from the machine. The machine I am running it on is Up Normal but the remote machine is always Down Normal. But both machines show that their status is Up Normal. The machine I run the command from lists its address as the loopback address and the remote address as the 10.x.x.x type of internal enterprise LAN address. The remote machine always shows DN.

Please see the below screenshot:

Please let me know if I need to provide more information.

Thank-you for reading this.

Respectfully,

me

Answer 1

To others who may have googled this and landed here:

I ran into this issue while trying to setup a 2 node version 3.9 cluster. Each one of my hosts had two interfaces, each host connected to a gigabit switch (which is connected to other servers) and the hosts additionally connected directly to each other via a 10-gig SFP+ cable. The idea being that other servers could query the cluster via the gigabit switch but the clusters themselves could communicate via 10-gig.

After trying many different combinations of listen_address and broadcast_address and broadcast_rpc_address I managed to solve this by setting the following (based on a default cassandra.yaml):

listen_address: (private 10-gig IP)
#broadcast_address: keep this line commented out
rpc_address: 0.0.0.0
broadcast_rpc_address: (gigabit switch IP)

Running a nodetool status should show the private IPs and show they are both up. Connecting to one of the hosts via the Java driver (3.1.3) and calling getAllHosts will show both hosts as being available on their gigabit switch IP.

Answer 2

For me, removal of the problematic node using the 'nodetool' removenode, solved the issue.

Answer 3

Turn off the firewall on both machines and then try if you receive this error.

Or a more sophisticated approach would just mean opening certain ports critical for communication between the Cassandra nodes.

Some ports to watch out for are shown at this URL: http://docs.datastax.com/en/cassandra/2.2/cassandra/configuration/secureFireWall.html

:D

Answer 4

There are two ways to do this : (2nd one worked better for me)

1. You need Python to start your server. (If you run cqlsh in your command line it tells you that Python is missing).

   • Install Python
   • Set env variables (To PATH add ;..python/bin;..cassandra/bin)

   • Open cmd, run following command : cassandra

   • Run below command in new cmd :

   cassandra-cli

   connect localhost /9160;

   • Once connected you can create keyspace. Start new command prompt and run : cqlsh

2. Just install the Community Version : http://www.planetcassandra.org/cassandra/ Install the required version. For older versions go to Archived versions link in the Page.

   Let the installation start the service for you by default. Once installed, you can run below command : cqlsh

No need to run Cassandra explicitly in this case.