Cassandra not compacting sstables?

Question

nodetool cfstats shows me the following output:

Read Count: 746287
Read Latency: 8.772114064696291 ms.
Write Count: 135629
Write Latency: 0.052691931666531494 ms.
Pending Flushes: 0
    Table: graphindex
    ** SSTable count: 230 **
    Space used (live): 1532001
    Space used (total): 1532001
    Space used by snapshots (total): 0
    SSTable Compression Ratio: 0.8071848230527264
    Memtable cell count: 159436
    Memtable data size: 2609278
    Memtable switch count: 1
    Local read count: 746287
    ** Local read latency: 8.773 ms **
    Local write count: 135629
    Local write latency: 0.053 ms
    Pending flushes: 0
    Bloom filter false positives: 1122
    Bloom filter false ratio: 0.00000
    Bloom filter space used: 39312
    Compacted partition minimum bytes: 43
    Compacted partition maximum bytes: 20501
    Compacted partition mean bytes: 70
    Average live cells per slice (last five minutes): 320.3775491198426
    Maximum live cells per slice (last five minutes): 3183.0
    ** Average tombstones per slice (last five minutes): 7997.852040836836 **
    ** Maximum tombstones per slice (last five minutes): 27078.0 **

As you can see the number of sstables is quite large. The table uses default compaction SizeTieredCompactionStrategy with min threshold 4 and max 32.

My questions are:

1. Why there are still so many sstables taking into account that the amount of data in the node is not large and the sstables are quite small? How (when) could this happen?

2. When SizeTieredCompactionStrategy actually triggers the compaction? In the other post I found that:

By default, a minor compaction can begin any time Cassandra creates four SSTables on disk for a column family. A minor compaction must begin before the total number of SSTables reaches 32.

But what should I do if the number of sstables already exceeds 32? Is running major compaction manually the only solution?

The reason why I'm asking is that because of big number of tombstones (last lines in the output above) and sstables, the read latency is becoming quite bad. The gc_grace_period is kept at low value, but since Cassandra didn't compact the sstables, the tombstones are still there. Or am I missing something?

Answer 1

With SizeTieredCompactionStrategy, it will only compact SSTables which are similar in size.

The problem is when you have lots of SSTables with differing sizes, they will not get picked up as candidates for compaction.

Be careful when running a manual compacting in STCS because you could end up with disproportionately large SSTables which will never get compacted again since it will not have a similar-sized partner, or it may take a long time until another similarly sized SSTable comes along.

Answer 2

I am investigating a similar issue I am having. There is this ticket in cassandra issue tracking.

Ok, this happens when cassandra decides to redistribute index summaries, by default every 60 minutes. Working on a fix, but in the mean time this can be avoided by setting index_summary_resize_interval_in_minutes to -1 in cassandra.yaml to disable this feature.

Testing this, will post the results.

Answer 3

Why there are still so many sstables taking into account that the amount of data in the node is not large and the sstables are quite small? How (when) could this happen? - This could happen especially when sstables are very small in size. When minor compaction is run, all sstables less than min_sstable_size (50mb default) will be placed in same bucket. When bucket is considered for compaction sstables upto max_threshold (default 32) will be considered for compaction and rest will be left alone. So for your data, if all 230 sstables are pretty small, only 32 will be considered for compaction with every minor gc.

If compaction is not triggering, you might have auto compaction turned of. You can alter table through CQL by changing compaction parameters. For example,

ALTER TABLE table1 WITH compaction = {'class': 'SizeTieredCompactionStrategy', 'enabled': true} ; 

All that said, I would first investigate why are so many small sstables created. Either your memtable or commitlog size is set to a small value or somehow flush is getting called too soon.