Cassandra - Read after a write by a different thread returns no data

Question

I have a 3 node cassandra cluster. A job is writing to a particular node and i try to read the data in a separate thread which might end up with a different cassandra node. I get no data from cassandra even the data by a separate thread inserts it before 10-15 seconds of a query by a different thread.

My replication factor is as below.

SELECT * FROM system.schema_keyspaces;

keyspace_name      | durable_writes | strategy_class                                       | strategy_options
--------------------+----------------+------------------------------------------------------+----------------------------
            application |           True | org.apache.cassandra.locator.NetworkTopologyStrategy |                {"DC1":"3"}
 system_distributed |           True |          org.apache.cassandra.locator.SimpleStrategy | {"replication_factor":"3"}
             system |           True |           org.apache.cassandra.locator.LocalStrategy |                         {}

Consistency :-

Read and Write is happening by a separate application.
Application which writes data has set the consistency level to ANY.
Application which reads data has set the consistency level to QUORUM.

Is there any configuration I am missing here?

What is the RF and what are the Consistency levels of the write and read requests? — RussS

Shlomi Livne Shlomi Livne · Accepted Answer · 2017-01-15T10:33:45

15 seconds is a very long time - so you should see the data.

Since you are writing with ANY (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/dml/dmlConfigConsistency.html) there is a chance that your cluster s not "healthy" and no replica node exists / or you have connectivity issues and writes will still succeed since they are written to hinted handoff.

To check this out you can:

Change write consistency to LOCAL_ONE/ONE - this will assure that at least one replica has received the data.
Enable probabilistic tracing (http://docs.datastax.com/en/archived/cassandra/3.x/cassandra/tools/toolsSetTraceProbability.html) and check if writes are delivered.

Is the write thread using batching or writing each row independently - If you are not using batching make sure you are using the token aware policy (http://docs.datastax.com/en/drivers/java/3.0/com/datastax/driver/core/policies/TokenAwarePolicy.html) and prepared statements - that will assure that the client will write to a node that has a replica.

Cassandra - Read after a write by a different thread returns no data

1 Answers