cassandra tombstones crossing threshold

Question

In the logs I see tombstone warning threshold.

Read 411 live rows and 1644 tombstone cells for query SELECT * FROM ks.tbl WHERE key = XYZ LIMIT 5000 (see tombstone_warn_threshold)

This is Cassandra 3.11.3, I see there are 2 sstables for this table and the partition XYZ exists in only one file. Now I dumped this sstable into json using sstabledump. I extracted the data of only this partition and I see there are only 411 rows in it. And all of them are active/live records, so I do not understand from where these tombstone are coming from?

This table has collection columns and there are cell tombstones for the collection columns when they were inserted. Does collection cell tombstones get counted as tombstones cells in the warning displayed?

Did a small test to see if collection tombstones are counted as tombstones and it does not seem so. So wondering where are those tombstones coming from in my above query.

CREATE TABLE tbl (
    col1 text,
    col2 text,
    c1 int,
    col3 map<text, text>,
    PRIMARY KEY (col1, col2)
) WITH CLUSTERING ORDER BY (col2 ASC)

cassandra@cqlsh:dev_test> insert into tbl (col1 , col2 , c1, col3 ) values('3','3',3,{'key':'value'});
cassandra@cqlsh:dev_test> select * from tbl where col1 = '3';
 col1 | col2 | c1 | col3
----------------+----------+----+------------------
              3 |        3 |  3 | {'key': 'value'}
(1 rows)

Tracing session: 4c2a1894-3151-11e9-838d-29ed5fcf59ee
 activity                                                                                 | timestamp                  | source        | source_elapsed | client
------------------------------------------------------------------------------------------+----------------------------+---------------+----------------+-----------
                                                                       Execute CQL3 query | 2019-02-15 18:41:25.145000 | 10.216.1.1 |              0 | 127.0.0.1
                  Parsing select * from tbl where col1 = '3'; [CoreThread-3]              | 2019-02-15 18:41:25.145000 | 10.216.1.1 |            177 | 127.0.0.1
                                                       Preparing statement [CoreThread-3] | 2019-02-15 18:41:25.145001 | 10.216.1.1 |            295 | 127.0.0.1
                                        Reading data from [/10.216.1.1] [CoreThread-3]    | 2019-02-15 18:41:25.146000 | 10.216.1.1 |            491 | 127.0.0.1
                                Executing single-partition query on tbl [CoreThread-2]    | 2019-02-15 18:41:25.146000 | 10.216.1.1 |            770 | 127.0.0.1
                                              Acquiring sstable references [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 |            897 | 127.0.0.1
 Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 |           1096 | 127.0.0.1
                                 Merged data from memtables and 1 sstables [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 |           1235 | 127.0.0.1
                                    Read 1 live rows and 0 tombstone cells [CoreThread-2] | 2019-02-15 18:41:25.146000 | 10.216.1.1 |           1317 | 127.0.0.1
                                                                         Request complete | 2019-02-15 18:41:25.146529 | 10.216.1.1 |           1529 | 127.0.0.1
[root@localhost tbl-8aaa6bc1315011e991e523330936276b]# sstabledump aa-1-bti-Data.db 
[
  {
    "partition" : {
      "key" : [ "3" ],
      "position" : 0
    },
    "rows" : [
      {
        "type" : "row",
        "position" : 41,
        "clustering" : [ "3" ],
        "liveness_info" : { "tstamp" : "2019-02-15T18:36:16.838103Z" },
        "cells" : [
          { "name" : "c1", "value" : 3 },
          { "name" : "col3", "deletion_info" : { "marked_deleted" : "2019-02-15T18:36:16.838102Z", "local_delete_time" : "2019-02-15T18:36:17Z" } },
          { "name" : "col3", "path" : [ "key" ], "value" : "value" }
        ]
      }
    ]
  }```

What is the table structure and how you perform updates/deletes? — Alex Ott
It has collections in it and updates/deletes do not happen much, just inserts. — nmakb

Alex Ott Alex Ott · Accepted Answer · 2019-02-22T14:45:15

If you insert collection data (map/list/set) with the same primary key, Cassandra doesn't know if the previous data exists or not, and simply inserts a tombstone to prevent accidental merge with previous version. The same happen if you update full collection instead of update operation on collection. More information you can find in following blog posts (1, 2).

If you don't need to perform partial updates of the collections, then it's always better to use frozen collections:

if you update/replace them, the tombstones aren't generated for them
they are more effectively saved on disk & read from it.

cassandra tombstones crossing threshold

1 Answers