Background: My application is running on a cluster system(which have 4 nodes), and system time of these four nodes are synchronizing by NTP. I use Write.QUORUM and Read.QUORUM strategy. The probability of this problem is not very high. Cassandra version is 1.0.3, I have tried Cassandra 1.1.1, this problem is still exist.
Problem: I deleted a column, but after 6 seconds, Cassandra can still get the old record which "isMarkedForDelete" is still false.
Is anybody meet the same problem? And how to solve it?
Detail: See the log below:
Node 3(Local node):
[pool-2-thread-42] 2012-06-27 14:49:23,732 SliceQueryFilter.java (line 123) collecting 0 of 2147483647: SuperColumn(667072 [......7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,......)
[pool-2-thread-44] 2012-06-27 14:51:21,367 StorageProxy.java (line 172) Mutations/ConsistencyLevel are [RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])])]/QUORUM
-- I delete this record at 14:51:21,367
[pool-2-thread-37] 2012-06-27 14:51:27,400 SliceQueryFilter.java (line 123) collecting 0 of 2147483647: SuperColumn(667072 [......,7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,......)
-- But I can still get the old record at 14:51:27,400
Node2:
[MutationStage:118] 2012-06-27 14:51:21,373 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])])
[MutationStage:118] 2012-06-27 14:51:21,374 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])]) applied. Sending response to 6692098@/192.168.0.3
[MutationStage:123] 2012-06-27 14:51:27,405 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [......,7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,......])
[MutationStage:123] 2012-06-27 14:51:27,405 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [......,7fffffffffffffff000001382ca96c8b636b698a:false:36@1340779097312016,.......]),])]) applied. Sending response to 6698516@/192.168.0.3
Node1:
[MutationStage:98] 2012-06-27 14:51:24,661 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779881338000,]),])])
[MutationStage:98] 2012-06-27 14:51:24,675 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a: true :4@1340779881338000,]),])]) applied. Sending response to 6692099@/192.168.0.3
[MutationStage:93] 2012-06-27 14:51:40,932 RowMutationVerbHandler.java (line 48) Applying RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779900915004,]),])])
DEBUG [MutationStage:93] 2012-06-27 14:51:40,933 RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc', key='3332', modifications=[ColumnFamily(fpr_index [SuperColumn(667072 [7fffffffffffffff000001382ca96c8b636b698a: true :4@1340779900915004,]),])]) applied. Sending response to 6706555@/192.168.0.3
[ReadStage:55] 2012-06-27 14:51:43,074 SliceQueryFilter.java (line 123) collecting 0 of 5000: 7fffffffffffffff000001382ca96c8b636b698a:true:4@1340779900915004
Node 4:
There is no log about this record on Node 4.
nodetool ringsay? – DNA