2
votes

I have a Cassandra cluster which is having gc_grace_seconds 10 days. auto compaction is enabled and running as per configuration but I am suspecting that auto compaction is not clearing the tombstones which are expired gc_grace_seconds duration(10 days). I am planning to run a major compaction on that table so my questions are.

1) Should I run major compaction without changing gc_grace_seconds 10 days?

2) Should I run major compaction changing gc_grace_seconds 0 days?

3) If I am changing gc_grace_seconds 0 so is it applicable for future data or already existing data with days gc_grace_seconds as well?

Thanks in advance.

2
What makes you want to clear tombstones, are you having a read latency issue because of them? i.e. treat the root cause not the symptom. Tombstones arent bad, just too many of them and incur performance problems - markc
Yes, we are facing read latency issue. - LetsNoSQL
Can you add to your question: table schema, sample cfstats output for this table, confirm if you see tombstone warning messages in the logs. Sample cqlsh trace. - markc
you might also want to have a look at this blog: academy.datastax.com/support-blog/… - markc

2 Answers

1
votes

First of all, you shouldn't set gc_grace_seconds to 0, unless on a single-node cluster. If gc_grace_seconds is set to some period, you must run repair at least once in every such period, otherwise you have a risk of data resurrection - which happens when one node on the cluster missed a deletion, and other nodes drop their tombstones, so a later repair will think the data is new and not realize it had been deleted already. If you ever set gc_grace_seconds to 0, any data which you previously deleted may be resurrected on the next repair, if the data happens to be on one of the replicas (because this specific replica missed the deletion because of some temporary problem).

So yes, the correct approach would be to run a major compaction with the original gc_grace_seconds of 10 days (and be sure to do a repair at least once every 10 days).

But you need to consider why you want to run a major compaction at all. Whether or not minor compaction can get rid of old (past 10 day) tombstones depends on a lot of factors, such as whether you recently made other modifications to the same partition that these tombstones sit in. But unless the tombstones are causing you major problems (tons of disk space, slower reads, etc.), it might not be worthwhile to do a major compaction. Major compaction is not free, and (at least in size-tiered compaction strategy) after it, all the data is sitting in one file and will take even longer until it can be compacted again.

1
votes

1) Should I run major compaction without changing gc_grace_seconds 10 days?

Yes. If set to 0 the tombstones will not be propagated to other nodes in the cluster. Which causes inconsistency in data.

3) If I am changing gc_grace_seconds 0 so is it applicable for future data or already existing data with days gc_grace_seconds as well?

If you change gc_grace_seconds it will be applicable to future data as well the current data.

If you want to clear tombstones by compacting I have two options for you

1) nodetool compact -s keyspace table

This will compact table and create sstables which are 50%-25%-12.5% and so on

2) nodetool compact --user-defined path/to/sstable

This will clear the tombstones from the sstable that is mentioned above.