Cassandra cfstats: differences between Live and Total used space values

2
votes

For about 1 month I'm seeing the following values of used space for the 3 nodes ( I have replication factor = 3) in my Cassandra cluster in nodetool cfstats output:

    Pending Tasks: 0
            Column Family: BinaryData
            SSTable count: 8145
            Space used (live): 787858513883
            Space used (total): 1060488819870

For other nodes I see good values, something like:

            Space used (live): 780599901299
            Space used (total): 780599901299

You can note a 25% difference (~254Gb) between Live and Total space. It seems I have a lot garbage on these 3 nodes which cannot be compacted for some reason. The column family I'm talking about has a LeveledCompaction strategy configured with SSTable size of 100Mb:

create column family BinaryData with key_validation_class=UTF8Type 
  and compaction_strategy=LeveledCompactionStrategy 
  and compaction_strategy_options={sstable_size_in_mb: 100};

Note, that total value staying for month on all of the three nodes. I relied Cassandra normalize data automatically.

What I tried to decrease space (without result):

  1. nodetool cleanup
  2. nodetool repair -pr
  3. nodetool compact [KEYSPACE] BinaryData (nothing happens: major compaction is ignored for LeveledCompaction strategy)

Are there any other things I should try to cleanup a garbage and free space?

3
cassandra
have you performed lots of deletion operation during this month time period? - abhi
I think yes, I dont have a precise value, it may be between 100Gb-1Tb of data have been deleted. But why only 3 nodes in my cluster have this issue? Why rest of nodes in the cluster have Live == Total? I'm using Cassandra 1.1.9 - odiszapc

3 Answers

1
votes

Ok, I have a solution. It looks like Cassandra issue. First, I went deep into the Cassandra 1.1.9 sources and noted that Cassandra perform some re-analysing of SStables during node starting. It removes the SStables marked as compacted, performs recalculation of used space, and do some other staff.

So, what I did is restarted the 3 problem nodes. The Total and Live values have become equals immediately after restart was completed and then Compaction process has been started and used space is reducing now.

0
votes

Leveled compaction creates sstables of a fixed, relatively small size, in your case it is 100Mb that are grouped into “levels”. Within each level, sstables are guaranteed to be non-overlapping. Each level is ten times as large as the previous.

So basically from this statement provided in cassandra doc, we can conclude that may be in your case ten time large level background is not formed yet, resulting to no compaction.

Coming to second question, since you have kept the replication factor as 3, so data has 3 duplicate copies, for which you have this anomaly.

And finally 25% difference between Live and Total space, as you know its due over deletion operation.

0
votes

For LeveledCompactionStrategy you want to set the sstable size to a max of around 15 MB. 100MB is going to cause you a lot of needless disk IO and it will cause it to take a long time for data to propagate to higher levels, making deleted data stick around for a long time.

With a lot of deletes, you are most likely hitting some of the issues with minor compactions not doing a great job cleaning up deleted data in Cassandra 1.1. There are a bunch of fixes for tombstone cleanup during minor compaction in Cassandra 1.2. Especially when combined with LCS. I would take a look at testing Cassandra 1.2 in your Dev/QA environment. 1.2 does still have some kinks being ironed out, so you will want to make sure to keep up to date with installing new versions, or even running off of the 1.2 branch in git, but for your data size and usage pattern, I think it will give you some definite improvements.