org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable inside kubernetes environment

Question

We have a setup wherein, one ignite server node serves 15 to 20 thick client nodes and 40 to 50 thin client nodes, thin client connection is singlton, In operation, some times we get below error,

org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable [sock=Socket[addr=hostnm19.hostx.com/10.13.10.19,port=30519,localport=57552]]

On the Server node, we are inserting data inside a third party store using CacheStoreAdapters

Don't know where it goes wrong since out of 100 operations one operation fails with the above error. Also, let me know what can we do for this failure handling.

Apache Ignite version: 2.8

Edits: (Code Snippet)

ClientConfiguration cfg = new ClientConfiguration()
                .setAddresses("host:port");

IgniteClient client = Ignition.startClient(cfg);   // this client is singleton

client.getOrCreateCache("ABC_CACHE").put(key, val);

StatckTrace:

org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable [sock=Socket[addr=hostnm19.hostx.com/10.13.10.19,port=30519,localport=57552]]
at org.apache.ignite.internal.client.thin.TcpClientChannel.handleIOError(TcpClientChannel.java:499)
at org.apache.ignite.internal.client.thin.TcpClientChannel.handleIOError(TcpClientChannel.java:491)
at org.apache.ignite.internal.client.thin.TcpClientChannel.access$100(TcpClientChannel.java:92)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.read(TcpClientChannel.java:538)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.readInt(TcpClientChannel.java:572)
at org.apache.ignite.internal.client.thin.TcpClientChannel.processNextResponse(TcpClientChannel.java:272)
at org.apache.ignite.internal.client.thin.TcpClientChannel.receive(TcpClientChannel.java:234)
at org.apache.ignite.internal.client.thin.TcpClientChannel.service(TcpClientChannel.java:171)
at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:160)
at org.apache.ignite.internal.client.thin.ReliableChannel.request(ReliableChannel.java:187)
at org.apache.ignite.internal.client.thin.TcpIgniteClient.getOrCreateCache(TcpIgniteClient.java:114)

Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.ignite.internal.client.thin.TcpClientChannel$ByteCountingDataInput.read(TcpClientChannel.java:535)
... 36 more

Unfortunately there's too few details. Is this port indeed available to connect on that host, from your current one? — alamar
@alamar all operations are working fine, but few get above error. Yes, the port is available. Let me know what details you need.. One which gets errors are same operations which work afterwards, even if no changes done. — iamLalit
Does it have a stack trace? There should be exception and path through Ignite client code. — alamar

alamar alamar · Accepted Answer · 2020-05-13T08:03:36

You probably have network or NAT configured which will reset connections when not used, or even sporadically. In this case, you will have to reconnect.

Another option, are you sure you are connecting to thin client port and not some other port?

org.apache.ignite.client.ClientConnectionException: Ignite cluster is unavailable inside kubernetes environment

1 Answers