I'm trying to emulate hinted handoff using Cassandra cluster in Docker.
Hinted handoff is active:
root@2f5aa8d649e2:/# nodetool statushandoff
Hinted handoff is running
The keyspace has a replication factor of 3:
cqlsh> DESCRIBE KEYSPACE imdb;
CREATE KEYSPACE imdb WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '2', 'dc2': '1'} AND durable_writes = true;
Then I shut one node down, turn on tracing and insert a new row:
cqlsh:imdb> insert into movies_by_actor (actor, movie_id, character, movie_title, salary) values ('TomHanks', uuid(), 'Character', 'Title', 1000);
Tracing session: e4a2cc20-42ce-11e7-bd49-cf534e0135c6
activity | timestamp | source | source_elapsed | client
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+------------+----------------+-----------
Execute CQL3 query | 2017-05-27 11:23:22.466000 | 172.13.0.2 | 0 | 127.0.0.1
Parsing insert into movies_by_actor (actor, movie_id, character, movie_title, salary) values ('TomHanks', uuid(), 'Character', 'Title', 1000); [Native-Transport-Requests-1] | 2017-05-27 11:23:22.467000 | 172.13.0.2 | 364 | 127.0.0.1
Preparing statement [Native-Transport-Requests-1] | 2017-05-27 11:23:22.467000 | 172.13.0.2 | 727 | 127.0.0.1
Determining replicas for mutation [Native-Transport-Requests-1] | 2017-05-27 11:23:22.468000 | 172.13.0.2 | 1354 | 127.0.0.1
Sending MUTATION message to /172.13.0.3 [MessagingService-Outgoing-/172.13.0.3-Small] | 2017-05-27 11:23:22.468000 | 172.13.0.2 | 1722 | 127.0.0.1
Sending MUTATION message to /172.13.0.6 [MessagingService-Outgoing-/172.13.0.6-Small] | 2017-05-27 11:23:22.468000 | 172.13.0.2 | 1722 | 127.0.0.1
MUTATION message received from /172.13.0.2 [MessagingService-Incoming-/172.13.0.2] | 2017-05-27 11:23:22.469000 | 172.13.0.3 | 30 | 127.0.0.1
MUTATION message received from /172.13.0.2 [MessagingService-Incoming-/172.13.0.2] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 35 | 127.0.0.1
Appending to commitlog [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.3 | 294 | 127.0.0.1
Appending to commitlog [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 292 | 127.0.0.1
Adding to movies_by_actor memtable [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 486 | 127.0.0.1
Enqueuing response to /172.13.0.2 [MutationStage-1] | 2017-05-27 11:23:22.469000 | 172.13.0.6 | 660 | 127.0.0.1
REQUEST_RESPONSE message received from /172.13.0.3 [MessagingService-Incoming-/172.13.0.3] | 2017-05-27 11:23:22.470000 | 172.13.0.2 | 3659 | 127.0.0.1
Processing response from /172.13.0.3 [RequestResponseStage-2] | 2017-05-27 11:23:22.470000 | 172.13.0.2 | 3820 | 127.0.0.1
Sending REQUEST_RESPONSE message to /172.13.0.2 [MessagingService-Outgoing-/172.13.0.2-Small] | 2017-05-27 11:23:22.472000 | 172.13.0.6 | 3533 | 127.0.0.1
REQUEST_RESPONSE message received from /172.13.0.6 [MessagingService-Incoming-/172.13.0.6] | 2017-05-27 11:23:22.473000 | 172.13.0.2 | 34 | 127.0.0.1
Processing response from /172.13.0.6 [RequestResponseStage-3] | 2017-05-27 11:23:22.473000 | 172.13.0.2 | 523 | 127.0.0.1
Request complete | 2017-05-27 11:23:22.469919 | 172.13.0.2 | 3919 | 127.0.0.1
As seen from the log, coordinator node 172.13.0.2 processes the request and contacts nodes 172.13.0.3 and 172.13.0.6. I would expect node 172.13.0.2 to save a hinted handoff, since the third node is unavailable. But when I check the system.hints table, it is empty:
cqlsh:imdb> select * from system.hints;
target_id | hint_id | message_version | mutation
-----------+---------+-----------------+----------
(0 rows)
The consistency level is set to default ONE. Could someone explain where the hinted handoff is stored, if at all?
