Apache Ignite node segmented

2
votes

From time to time I keep getting node segmented. It happens in a cluster with ~40 nodes - it happens only on one node at a time. A few times it happened while there was some heavy GC work going on. On the other hand I have seen that similar heavy GC work going on and no node got segmented. I have tuned failure detection timeout to be bigger than max GC I was experiencing but that didn't help - failureDetection timeout is almost 2x bigger than max GC. How can I figure out if this is really GC or network issues? I doubt it is related to networking as other nodes would fail as wel. When the process gets restarted it works fine so I would rule out network issues. Where can I look at the code which produces EVT_NODE_SEGMENTED? I debugged the IgniteConfiguration object and saw that segResolvers are null/empty so I have no clue where the event it published.

S3 based discovery is used, not sure it matters here(ignite 1.9).

I wonder under which conditions such event is produced? Unable to connect to majority of other nodes or all of them?

1
ignite
Do you have any other timeouts configured? I mean networkTimeout, connectTimeout, etc. Also, it will be helpful to take a look at your configuration and logs from all nodes. - Evgenii Zhuravlev
I have only configured failureDetectionTimeout and slowClientQueueLimit. I can sometimes see warnings related to socketWriteTimeout - that a timeout occurred and that I should probably increase it. - Bart
All the other nodes logs contain entry that the specific node FAILED. [disco-event-worker-#28%null%] WARN o.a.i.i.m.d.GridDiscoveryManager - Node FAILED: TcpDiscoveryNode [id=6d74ee33-0068-4e4f-8e67-b8f2a8bca05b, addrs=[0:0:0:0:0:0:0:1%lo, 10.50.9.176, 127.0.0.1], sockAddrs=[/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, ip-10-50-9-176.ec2.internal/10.50.9.176:47500], discPort=47500, order=74, intOrder=57, lastExchangeTime=1506065144854, loc=false, ver=1.9.0#20170302-sha1:a8169d0a, isClient=false] - Bart
While on the node that died I can find [tcp-disco-msg-worker-#2%null%] WARN o.a.i.s.d.tcp.TcpDiscoverySpi - Node is out of topology (probably, due to short-time network problems). - Bart

1 Answers

0
votes

This event occurs when node disconnects and can't connect back, see ClientImpl.java and ServerImpl.java

look at logs at segmented node, it must be something like Node is out of topology (probably, due to short-time network problems message, so you can figure out exact problem.