How can the Cassandra commitlog be corrupted?

Question

This is the second time my commitlog is corrupted, and the server refuses to start. What worries me is that I get these error issues even if no update were made to the database.

My config says that commitlog are synced every 10s seconds, so how can a file be corrupt unless a crash occurs within these 10 seconds?

Is this a Cassandra bug? Or by design, i.e. bad design?

I am using 3.4 on Windows 10, Datastax installer.

In the stdout log, the last part is

INFO  06:17:39 Replaying C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812251.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471411951134.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471454506802.log, C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471532812678.log
ERROR 06:17:39 Exiting due to error while processing commit log during initialization.
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Could not read commit log descriptor in file C:\Program Files\DataStax-DDC\data\commitlog\CommitLog-6-1471353812252.log
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:611) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:373) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:236) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:192) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:172) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:283) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.4.0.jar:3.4.0]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.4.0.jar:3.4.0]

Hi, were you able to debug this issue. we are facing the same error . I know its an old question. But were you able to fix it ? — Raveesh Sharma
No, and I didn't get any answer in cassandra jira. I think they want me to use a plain cassandra version, not datastax. My plan is to do that when I come nearer deploy. — mattias
Most likely, it is the same problem as this issues.apache.org/jira/browse/CASSANDRA-11995, which is fixed in Cassandra 3.11 — mattias

Hemalatha Hemalatha · Accepted Answer · 2017-02-17T08:15:18

I have seen similar errors. This happens, when Cassandra process gets crahed may be due to OOM. Run "dmesg" and check if it was killed due to OOM. In that case there is possibility that commit log it was writing to was corrupted or its of 0kb file (check size of above file in error), and it throws the above error when Cassandra is restarted and it replays that file.

How can the Cassandra commitlog be corrupted?

1 Answers