I have a Cassandra Cluster conformed with the nodes
{ 192.168.120.57, 192.168.120.58, 192.168.120.59 }, with replication factor of 2
I detected the node 192.168.120.59 was behaving irregularly
-> The service was running (sudo service cassandra status)
-> The nodetool also told me that the node was running
XXXX@cassandra-prod03:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 192.168.120.57 4.05 GB 256 ? 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb rack1
UN 192.168.120.58 4.4 GB 256 ? 53bc77ff-50b3-4bf0-bd55-17e2bb7c8208 rack1
UN 192.168.120.59 2.77 GB 256 ? 03d8d7fd-1d86-4034-a537-9915adb0d4b3 rack1
BUT -> I couldn't connect by DevCenter (the last time I tried I could and the cassandra.yml hasn't changed)
I looked into the log of this node (192.168.120.59), and showed:
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,827 HintsDispatchExecutor.java:224 - Failed to dispatch hints file 6d0f5961-a600-42c2-8ba5-5b6ebff52ceb-1465184315371-1.hints: file is corrupted ({})
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,830 CassandraDaemon.java:195 - Exception in thread Thread[HintsDispatcher:1,1,main]
ERROR [HintsDispatcher:1] 2016-06-23 16:57:01,834 StorageService.java:470 - Stopping gossiper
ERROR [HintsDispatcher:1] 2016-06-23 16:57:03,842 StorageService.java:480 - Stopping native transport
In the other nodes, the log showed many many times:
DEBUG [GossipTasks:1] 2016-06-24 07:56:21,551 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:22,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:23,552 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:24,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:25,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:26,553 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
DEBUG [GossipTasks:1] 2016-06-24 07:56:27,554 Gossiper.java:336 - Convicting /192.168.120.59 with status shutdown - alive false
These are the things I have done and which didn't work:
1) reboot the machine.
2) nodetool repair.
XXXXs@cassandra-prod03:~$ nodetool repair
[2016-06-23 17:07:37,582] Starting repair command #1, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
Jun 23, 2016 5:53:57 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check the connection: java.net.SocketTimeoutException: Read timed out
Exception occurred during clean-up. java.lang.reflect.UndeclaredThrowableException
error: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: [2016-06-23 18:06:22,229] JMX connection closed. You should check server log for repair status of keyspace recommendersjobs(Subsequent keyspaces are not going to be repaired).
at org.apache.cassandra.tools.RepairRunner.handleConnectionFailed(RepairRunner.java:97)
at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:86)
at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:275)
at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:352)
at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:337)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
at javax.management.remote.rmi.RMIConnector.sendNotification(RMIConnector.java:441)
at javax.management.remote.rmi.RMIConnector.access$1200(RMIConnector.java:121)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
at javax.management.remote.rmi.RMIConnector$RMINotifClient.fetchNotifs(RMIConnector.java:1352)
3) nodetool scrub.
After this I followed this page https://docs.datastax.com/en/cassandra/1.2/cassandra/operations/ops_backup_noderestart_t.html
On the node that presented the irregular behaviour:
1) Deleted all tables
2) Copy and Paste the tables that were in the snapshot sub-directory
3) Restarted cassandra service (sudo service cassandra start)
Now the Dev-Center can connect
4) I Ran nodetool repair again, and it seems that is not finishing. It has been running for more than 3 hours and is still in 0% (but for now didn't throw the other error), but I think it is not working.
XXXX@cassandra-prod03:/var/lib/cassandra/data/recommendersjobs$ nodetool repair
[2016-06-24 12:24:13,901] Starting repair command #2, repairing keyspace recommendersjobs with repair options (parallelism: parallel, primary range: false, incremental: true, job threads: 1, ColumnFamilies: [], dataCenters: [], hosts: [], # of ranges: 512)
[2016-06-24 12:24:17,235] Repair session 7990e900-3a30-11e6-9861-e998758604bf for range [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]] failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser, [(-4211712190859502280,-4203992560184719114], (-8318576077833367495,-8293862912754480840], (6846385493332587078,6848188842228182830], (-6460899805527117383,-6451580681320322560], (-927549969226058129,-890279568362794135], (-6984061332646781425,-6971866923830433843], (4166814791483192953,4181831925595728309], (212986607214778070,288868023576418487], (656453115636318699,670358990995510048], (-3857111065907140708,-3844748631754416189], (-731643009820930522,-708535538994856284], (-3646398519882341977,-3630383383747491969], (-4215258070829115921,-4211712190859502280], (6972519766285293067,6983272857539313387], (8698322395242654268,8707318407090795417], (-1229710138608978352,-1142107267252652523], (-6043663680829718954,-6000947666680635781], (-2555062915045199657,-2518876915835081063], (700476075055800053,714305291929584762], (-9040157763677832405,-9035752029446223323], (-5508447172120753932,-5486202973508763691], (685259738113005224,700476075055800053], (-3498127715533695144,-3481009398659298196], (-6642818648542557724,-6636310914487378937], (8434600097698749124,8447388029675461333], (-9196939878235591876,-9176601682787902008], (-8392736270165548628,-8358469817741800124], (2668692603556301942,2674434559513729415], (-1868370051056061493,-1860700618558449042], (-6090102443758865946,-6081413019910708044], (-1234056662103299181,-1233526281479424892], (5500764053339166116,5524556889632227041], (-2343395791776959523,-2315566773401476532], (-1401161284995746969,-1393787270658712827], (-2204381944261735990,-2170477179223028079], (6158419946524545293,6171118529963906232], (-4713529768344258113,-4662557763507006574], (-6844075271953818452,-6821121360440227129], (-2945271950056514025,-2943444578534843676], (336935533896784347,357900664320554816], (-3228825867554770158,-3208435083108021414], (-4955223362385024239,-4912831044277064162], (4255028482041276438,4266791030997837989], (6476681916885657705,6500903072598398827], (-1233526281479424892,-1229710138608978352], (1478830670694875260,1516022971169730278], (5055515752742518226,5080231569691472194], (-8220031638992810484,-8180158125318738371], (-1644262187188908209,-1640818085920683524], (-4203992560184719114,-4201062514351678359], (-5222304299920694120,-5219839223573200521], (-8773724135260117843,-8764792082520695033], (-1355888553876954583,-1337212328034401607], (4736203535340725934,4798689233015535742], (1271210285313194340,1296150437356677365], (-3540891434049035738,-3498615337040509123], (-6469678775729021799,-6460899805527117383], (6720989190599823387,6725081429318018351], (-3498615337040509123,-3498127715533695144], (-3146863944770153663,-3146145879742664566], (1516022971169730278,1543205759172249070], (2435469497781312092,2442862581422161803], (-6451580681320322560,-6450897957558236116], (6401972194395279212,6422279168069398956], (-3185403021559141625,-3146863944770153663], (714305291929584762,737464206890275575], (-8358469817741800124,-8348341616724830280], (1175374807063094969,1214972878391460620], (-7604581392807650182,-7599009809842308024], (737464206890275575,753307058766478061], (6142712373291735013,6158419946524545293], (-4780493053770543287,-4715780574795279713], (-6475595045987094984,-6472782860721811368], (-6000947666680635781,-5984299660747839008], (1296150437356677365,1341203791801750057], (-708535538994856284,-698490751868803865], (7377888475905026189,7453466617724696329], (-6634208348348544983,-6608369916989089358], (6422279168069398956,6425564550565269465], (7864684753798709404,7865351483298068861], (-1142107267252652523,-1130763472498460650], (2661263629361374231,2668692603556301942], (5080231569691472194,5114813438584055434], (-3146145879742664566,-3133020507157500012], (-8847809439930946447,-8773724135260117843], (-7455563937756770941,-7393173741279261649], (-8001929233768101133,-7997359338916453120], (-3208435083108021414,-3185403021559141625], (-8504346858359312144,-8489394145547730314], (330577967974463458,336935533896784347], (7563807678381276197,7582113707662028389], (-8542082760751850432,-8504346858359312144], (-3554430985922725406,-3540891434049035738], (-5146821899868165131,-5132879094583817494], (-240050552784290962,-235772081518906038], (3192195406141461438,3232311876000191165], (4396701296202861119,4435447637968520477], (-6608369916989089358,-6589281199205137237], (-1981758794620356502,-1980394476029734464], (-3563262972235992519,-3554430985922725406], (-8237266126949227268,-8229088751363770412], (4603382773438618005,4650956890538380770], (7199369658829377994,7213887142216841811], (-4962133286795414966,-4960837410336448181], (5732306214090585629,5737923082195078719], (-1640818085920683524,-1638198095283336424], (2953772832389323874,2968606959459466094], (-1235736831018577452,-1234056662103299181], (-8489394145547730314,-8465866933095448418], (-8764792082520695033,-8742184848088246957], (8970234525288431416,8995803891393002810], (-8465866933095448418,-8446751642929552235], (-4836950976239959831,-4798814403290155024], (8821435912538079613,8871870554368996448], (-6636310914487378937,-6634208348348544983], (1127798841363210583,1175374807063094969], (1618461143046543312,1632362492578265392], (4994543089928849457,5055515752742518226], (1593785714944278507,1618461143046543312], (-8885929495137049667,-8847809439930946447], (7514316870399738740,7535648178040640648], (-7997359338916453120,-7884947734877215590], (6848188842228182830,6851091943952222998], (-7468624498429238352,-7467528674261593018], (-5420066870415394025,-5415525963610970555], (-1990341934536435876,-1981758794620356502], (-2119663304358780223,-2025913005077454264], (6951976975129490336,6972519766285293067], (4814028481954927072,4845532697727637101], (6905718428902118395,6951976975129490336], (2442862581422161803,2473486929810000384], (-7050886265796431130,-6984061332646781425], (899736981222314462,902246412094263205], (4181831925595728309,4186043100749193555], (5228334028316552580,5251807687845401107], (7453466617724696329,7514316870399738740], (-5313682991153179320,-5303078638925764028], (4435447637968520477,4466559239870893508], (-1326922053645316997,-1235736831018577452], (1401641821520873210,1444010990722687070], (1751424608310627366,1764837197082917798], (4501052626153108190,4556808478342919483], (4927485308468803128,4934126082298511287], (9034829150987061085,9125279489658634069], (8296589981550156872,8314630209670832253], (8448913759582599761,8531687930406385678], (-6946999157236744448,-6844075271953818452], (1444010990722687070,1478830670694875260], (8959600358987559211,8960859570137987260], (-1336371416389083059,-1326922053645316997], (-61295809418728622,-42017489632420450], (8969118191184771962,8970234525288431416], (1635093197288200819,1646180075666173338], (4934126082298511287,4936412559469748364], (1751396675459233161,1751424608310627366], (-8742184848088246957,-8729944250195454990], (-8229088751363770412,-8220031638992810484], (4598535296351629260,4603382773438618005], (-8174261525446719402,-8163507145438773646], (755491933174209966,772199007723251143], (-8624222028448607604,-8600591663798592212], (-7519020387526209046,-7508903488301733990], (753307058766478061,755491933174209966], (7582113707662028389,7704299093463733587], (8962972620592857243,8969118191184771962], (-3630383383747491969,-3629090531220364683], (-4468127887710219501,-4455300706501649674], (5852098730437702158,5901360113812253268], (-744327910454436920,-731643009820930522], (1911291563147302352,1937039485053092324], (6233932117267819795,6310871518339058610], (-890279568362794135,-887061681980670889], (-3679929070886276718,-3646398519882341977], (670358990995510048,685259738113005224], (4845532697727637101,4882596066849804097], (-6472782860721811368,-6469678775729021799], (-4715780574795279713,-4713529768344258113], (3078864489691764074,3115136329005690845], (9125279489658634069,9133717713662800417], (8960859570137987260,8962972620592857243], (9203390852073803003,-9222340996132828621], (4798689233015535742,4814028481954927072], (-6821121360440227129,-6777406043019946301], (6310871518339058610,6322362350826191135], (-4233493116436171657,-4215258070829115921], (3130822610289703870,3145528043810254358], (5114813438584055434,5123797488219630832], (-6081413019910708044,-6079621320236638978], (8447388029675461333,8448913759582599761], (-3795970094034238564,-3779568630549582632], (-8180158125318738371,-8174261525446719402], (-3844748631754416189,-3795970094034238564], (4113442287147808270,4131576074431163736], (4936412559469748364,4994543089928849457], (4466559239870893508,4501052626153108190], (-3263191959146785290,-3228825867554770158], (6766473197422691050,6766938926557171347], (-8899258035651035854,-8885929495137049667], (-5381097520433537861,-5313682991153179320], (6883419270850869410,6905718428902118395], (5303695175174121453,5306682210025644500], (-9176601682787902008,-9096402285624130055], (-5219839223573200521,-5146821899868165131], (7862240594123780934,7864684753798709404], (6425564550565269465,6447403433784268090], (8996665254881958342,9032628945801968053], (-4912831044277064162,-4902449072491610799], (-2025913005077454264,-1990341934536435876], (-7467528674261593018,-7455563937756770941], (-3293906917083373209,-3283303577982631091], (4186043100749193555,4255028482041276438], (-4902449072491610799,-4875492513338002087], (8531687930406385678,8562083491872293504], (-8600591663798592212,-8567258569010210224], (-6079621320236638978,-6077962914827619257], (-4960837410336448181,-4955223362385024239], (-7634134348451706917,-7604581392807650182], (1632362492578265392,1635093197288200819], (291067515209041750,330577967974463458], (7789905572870286790,7790287256445162065], (-8007287567032110044,-8001929233768101133], (-6971866923830433843,-6946999157236744448], (9032628945801968053,9034829150987061085], (-4798814403290155024,-4780493053770543287], (3905716352976079268,3920453505268467479], (8995803891393002810,8996665254881958342], (-1393787270658712827,-1355888553876954583], (6522490073023565012,6553480210988421066], (7162495952820208409,7199369658829377994], (4882596066849804097,4927485308468803128], (-5893410924582496350,-5858193870200257506], (8233960782654491894,8252708412304213777], (6725081429318018351,6766473197422691050], (-5303078638925764028,-5222304299920694120], (2376875294823216717,2379477119284337609], (-7393173741279261649,-7376038212607696986], (-1337212328034401607,-1336371416389083059], (902246412094263205,902703964241874282], (288868023576418487,291067515209041750], (1341203791801750057,1364316712425392743], (5199663539610295629,5228334028316552580], (6447403433784268090,6476681916885657705], (-3481009398659298196,-3391281017675748608], (4266791030997837989,4315496651682018963], (3115136329005690845,3130822610289703870], (-7599009809842308024,-7598746015885846943]]] Validation failed in /192.168.120.58 (progress: 0%)
In this log appears:
1) failed with error [repair #7990e900-3a30-11e6-9861-e998758604bf on recommendersjobs/postulationsbyuser (near the middle)
2) Validation failed in /192.168.120.58 (progress: 0%) (in the last part)
While this repair is running (I think it is running), in the log of this node and in the others nodes of the cluster, this appears many times:
DEBUG [ReadRepairStage:4] 2016-06-24 14:08:18,318 ReadCallback.java:235 - Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-3297346077674417080, 42555f31313131303131333538) (d41d8cd98f00b204e9800998ecf8427e vs 9e7f1588ffcd53d13ab3f8bb2be0f05e)
at org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) ~[apache-cassandra-3.3.jar:3.3]
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:226) ~[apache-cassandra-3.3.jar:3.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
NOTE 1: The cassandra was and is working because of the replication factor of 2.
NOTE 2: The cassandra Cluster is being used in production, and applications are inserting and reading rows.
What can I do to correct this irregular behaviour?
I thought of creating a new node and swap it with the irregular node, what are the correct steps for doing this?
