Datastax Mismatch for Key Issue

2
votes

Our current setup contain DSE 5.0.2 version with 3 node cluster.Currently we are facing issue with heavy load and node failure issue.Debug.log details is given below:

DEBUG [ReadRepairStage:8] 2016-09-27 14:11:58,781 ReadCallback.java:234 - Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(5503649670304043860, 343233) (45cf191fb10d902dc052aa76f7f0b54d vs ffa7b4097e7fa05de794371092c51c68) at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:225) ~[cassandra-all-3.0.7.1159.jar:3.0.7.1159] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77]

1
datastaxdatastax-enterprisedatastax-java-driver

1 Answers

3
votes

I'm answering this from the perspective of what the error you posted means. However I dont think this alone will be the cause of your problems. Without seeing all the logs from the nodes in your cluster it is hard to tell.

The Digest mismatch you posted actually comes from a read repair. This docs link explains it at a high level (note contrary to what the doc says, read repair can be blocking at other CLs too):

https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsRepairNodesReadRepair.html

If you are seeing too many read repairs and you have multiple DCs you might want to consider setting read_repair_chance lower and increasing dclocal_read_repair_chance, by default iirc they are 0.1 and 0 respectively so not always the most optimal.

I have seen these cause read timeouts as a digest mismatch can cause a blocking read repair. Your best bet if you think this is causing problems is to either run the query in cqlsh with tracing or use probabilistic tracing to log queries which you can view traces on in retrospect

Doc links:

https://docs.datastax.com/en/cql/3.3/cql/cql_reference/tracing_r.html

https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsSetTraceProbability.html