Error with multi-node cassandra setup + connection

Question

So I'm trying to setup multi-node cassandra cluster.

I've setup 3 cassandra nodes on 3 different servers. In all of the three cassandra.yaml files, I've defined the same one seed node. After clearing the data and restarting the services one by one, I can see them showing up and the nodetool status shows UN for all entries. Now I use express-cassandra for connecting to the cassandra, and that works just fine, but it only works for the first few queries and afterwards, it just crashes with the following error -

Error during find query on DB -> ResponseError: Server failure during read query at consistency ONE (1 responses were required but only 0 replicas responded, 2 failed)

It does not matter on which read query it fails, but it's up for like 10 seconds and then it dies. Sometimes it dies on queries which were successful on previous run. Also, I'm just wondering, why it says that 2 failed, if there are 3 nodes (one seed node)?

I noticed, that express-cassandra notified that the Replication factor has been changed and to run the nodetool repair, but there I keep getting the following error message -

Validation failed in /xxx.xxx.xxx.xxx (progress: 0%) [2018-07-02 08:10:33,447] Some repair failed

Where xxx.xxx.xxx.xxx is one of the three node IP addresses. I tried to run the nodetool repair on each of the nodes, but I keep getting identical errors on each of the servers.

These are my express-cassandra setup properties (xxx.xxx.xxx.xxx is my seed ip address):

    clientOptions: {
      contactPoints: ['xxx.xxx.xxx.xxx'],
      protocolOptions: {
        port: 9042
      },
      keyspace: 'test_keyspace',
      queryOptions: {
        consistency: cassandra.consistencies.one
      },
      socketOptions: {
        readTimeout: 0
      }
    },
    ormOptions: {
      defaultReplicationStrategy: {
        class: 'NetworkTopologyStrategy',
        dc1: 3
      }
    }

I'm not sure if the NetworkTOpologyStrategy is correctly set up, but after starting up the express.js I ran a write query to the database, and it populated the data on all three nodes.

Any help on resolving these both errors would be wonderful!

1. do you see any error in system.log ? 2. Use Simple strategy — Laxmikant
@Laxmikant Why should I use simple strategy? Will it still work in this case, and will it give any performance improvements? Also - Do I have to change the Snitch then? — Patiss
it wont impact performance ..as far as snitch is concerned u can use GossipingPropertyFileSnitch — Laxmikant
There are no error logs in system.log. I managed to start it up successfully, but now I get the following error from time to time: ResponseError: Server timeout during read query at consistency ONE (0 replica(s) responded over 1 required). Now it seems that the cassandra database was more stable with 1 node, since it did not timeout. With 3 nodes it time outs every 40 seconds. Any guess? — Patiss

Patiss Patiss · Accepted Answer · 2019-04-25T09:06:43

Providing an answer here, just in case anyone else struggles.

The issue was related to the NetworkTopologyStrategy. I switched to SimpleStrategy, and it started to work.

I had just one DC.

Not sure, what exactly was causing the issue, but if anyone is aware, feel free to add it in comment, and I'll update the answer.

Error with multi-node cassandra setup + connection

1 Answers