Cassandra 3.0 one node is not getting sync with other 2 nodes in 3 node cluster

0
votes

WARN [GossipStage:1] 2020-03-25 10:34:52,849 Gossiper.java:1161 - received an invalid gossip generation for peer /10.x.x.x; local time = 1585157692, received generation = 1695958421 WARN [GossipStage:1] 2020-03-25 10:34:52,849 Gossiper.java:1161 - received an invalid gossip generation for peer /10.x.x.x; local time = 1585157692, received generation = 1695958421

/10.x.x.150 generation:0 heartbeat:0 TOKENS: not present /10.x.x.151 generation:1585155568 heartbeat:2488 STATUS:16:NORMAL,-1019803973446170140 LOAD:2438:2893850.0 SCHEMA:12:2075709a-c594-36b8-8708-3de7fb58325c RACK:10:cassandra_rack RELEASE_VERSION:4:3.11.0-E000 NET_VERSION:1:11 RPC_READY:28:true TOKENS:15: /10.x.x.149 generation:1695958421 heartbeat:2476 STATUS:16:NORMAL,-1083796634382577337 LOAD:2437:1221526.0 SCHEMA:12:2075709a-c594-36b8-8708-3de7fb58325c DC:8:BVI15 RACK:10:cassandra_rack RELEASE_VERSION:4:3.11.0-E000 NET_VERSION:1:11 RPC_READY:29:true TOKENS:15:

Please let me know how to resove it.

2
cassandra
Could you update your question with more info regarding your setup and environment configuration? It's near impossible to figure out what you're doing just from logs alone – Skwiggs
how long nodes are running? – Alex Ott
also look to that answer: stackoverflow.com/questions/58193695/… – Alex Ott

2 Answers

0
votes

Make sure that your nodes' system clocks are in sync by installing a properly functioning ntp server on each node. The warning you received occurs when the generation received from another node is more than one year after the node's local time. In the warning messages, the local generation appears to be 1585157692, which is 3/25/2020, whereas the generation received from the other node is 1695958421, which is 9/29/2023. The generation is initialized from the system clock when the node is started.

0
votes

Update to system.local did not work as it seems we are in higher version Cassandra 3.0.11..We finally had to wipe out all cassandra data in the cluster and restarted. After that We restored nodetool snapshot backup which I took before the data cleanup