Can a commitlog created several hours ago apply to a cassandra cluster?

1
votes

I have a cassandra cluster running in the kubernetes environment, in a namespace, say test1, and I want to test the restore function. So I did a snapshot in the test1 cassandra, moved the snapshot to another node, from these data started a cassandra cluster in another namespace, say test2. The problem was, test2 cassandra cluster replaced test1 cluster totally, Customer's data that should write to the test1 cassandra cluster had written to the test2 cassandra cluster.

An hour later, I noticed this, stopped test2 cassandra cluster, and restarted test1 cassandra cluster, although it has come back to work shortly, but some data was lost.

After a while, I noticed there was some commitlog at that period in the test2 cassandra node, and want to recover these data. Can I just stop the test1 cassandra cluster, put these commitlog files into the commitlog directory of test1 cassandra node, then start cassandra, let cassandra to replay these commitlog ?

1
cassandra

1 Answers

0
votes

Commitlogs from one node can’t be played on a different node or cluster. They are transactions specific to the node they came from.

source - read the notes ("Important note: A point-in-time restore requires a cluster restart for the commitlog replay to run" and "Some Helpful Notes for Planning")

Later update:

I'm not sure what you mean by "test2 cassandra cluster replaced test1 cluster totally". My assumption is that you restored everything, including system keyspaces.

If you did this, yes, then applying the commit logs might work since besides the IP's and the hostnames, the cluster is sort of the same.

If you look into the CommitLogReader code, you will see that a mutation is considered invalid if an UnknownTableException is thrown (basically if the id of the table is not the same between the commit log and the system keyspace).

I did a similar test on ccm and successfully replayed the commit logs after I changed the id of the table both on file system and in system_schema.tables.

From my perspective your cluster is pretty messed up. Although you could do this and it might work, I think you will always have a high risk of corrupt data.

So, since in the datastax documentation (which we could consider the base documentation for Cassandra) is stated that this operation is supported I am not recommending this.