How to force delete tombstone created with USING TIMESTAMP in delete CQL statement?

Question

We have a table which we deleted bunch of rows from using java max long (9223372036854775807) as timestamp. For example,

DELETE r_id FROM orderbook USING TIMESTAMP 9223372036854775807 WHERE o_id='' AND p_id='' AND e_id='' AND a_id='a1' AND ord_id = 645e7d3c-aef7-4e3c-b834-24b792cf2e55;

These tombstones are created in sstable with markedForDeleteAt = 9223372036854775807.

Sample output from sstable2json

[ {"key": ":::a1", "cells": [["645e7d3c-aef7-4e3c-b834-24b792cf2e51:_","645e7d3c-aef7-4e3c-b834-24b792cf2e51:!",9223372036854775807,"t",1476520163], ["645e7d3c-aef7-4e3c-b834-24b792cf2e52:","",1], ["645e7d3c-aef7-4e3c-b834-24b792cf2e55:","",1], ["645e7d3c-aef7-4e3c-b834-24b792cf2e55:r_id",1476520867,9223372036854775807,"d"]]} ]

Tombstones (range ("t") or otherwise ("d")) created with such high time isn't getting collected with minor or major compaction. We even tried setting gc_grace_seconds to 0 and run major compaction but no luck. I am thinking that 'markedForDeleteAt + gc_grace_seconds > compaction time' equation is playing out and that's why tombstones are not collected. But then I read cassandra code and it seems like localDeletionTime is considered in equation and not markedForDeleteAt.

 * The local server timestamp, in seconds since the unix epoch, at which this tombstone was created. This is
 * only used for purposes of purging the tombstone after gc_grace_seconds have elapsed.
 */
public final int localDeletionTime;

With all that, how can I force remove all tombstones from sstable?

user2903819 user2903819 · Accepted Answer · 2016-10-16T05:29:25

CASSANDRA-12792 - Due to Cassandra bug filled yesterday, it isn't possible to remove tombstones written with Long.MAX_VALUE with compaction. I had to do ETL and table truncate to get rid of tombstones.

In db/compaction/LazilyCompactedRow.java we only check for < MaxPurgeableTimeStamp eg: (this.maxRowTombstone.markedForDeleteAt < getMaxPurgeableTimestamp()) this should probably be <=

How to force delete tombstone created with USING TIMESTAMP in delete CQL statement?

1 Answers