0
votes

I'm starting the Hazelcast process on host 10.99.248.130, why is it connecting to an instance in host 10.99.248.138?

2020-10-22 21:14:29,124 [ServerService Thread Pool -- 84] INFO com.hazelcast.system - [10.99.248.130]:5701 [dev] [3.7.5] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved. 2020-10-22 21:14:29,124 [ServerService Thread Pool -- 84] INFO com.hazelcast.system - [10.99.248.130]:5701 [dev] [3.7.5] Configured Hazelcast Serialization version : 1 2020-10-22 21:14:29,797 [ServerService Thread Pool -- 84] INFO com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator - [10.99.248.130]:5701 [dev] [3.7.5] Backpressure is disabled 2020-10-22 21:14:31,243 [ServerService Thread Pool -- 84] INFO com.hazelcast.instance.Node - [10.99.248.130]:5701 [dev] [3.7.5] Creating TcpIpJoiner 2020-10-22 21:14:31,764 [ServerService Thread Pool -- 84] INFO com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl - [10.99.248.130]:5701 [dev] [3.7.5] Starting 2 partition threads 2020-10-22 21:14:31,773 [ServerService Thread Pool -- 84] INFO com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl - [10.99.248.130]:5701 [dev] [3.7.5] Starting 3 generic threads (1 dedicated for priority tasks) 2020-10-22 21:14:31,805 [ServerService Thread Pool -- 84] INFO com.hazelcast.core.LifecycleService - [10.99.248.130]:5701 [dev] [3.7.5] [10.99.248.130]:5701 is STARTING 2020-10-22 21:14:31,811 [ServerService Thread Pool -- 84] INFO com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel - [10.99.248.130]:5701 [dev] [3.7.5] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads 2020-10-22 21:14:31,905 [hz.SMP.cached.thread-3] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Connecting to /10.99.248.138:5702, timeout: 0, bind-any: true 2020-10-22 21:14:31,910 [hz.SMP.cached.thread-3] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Could not connect to: /10.99.248.138:5702. Reason: SocketException[Connection refused to address /10.99.248.138:5702] 2020-10-22 21:14:31,910 [hz.SMP.cached.thread-3] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [10.99.248.130]:5701 [dev] [3.7.5] [10.99.248.138]:5702 is added to the blacklist. 2020-10-22 21:14:31,914 [hz.SMP.cached.thread-2] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Connecting to /10.99.248.138:5703, timeout: 0, bind-any: true 2020-10-22 21:14:31,915 [hz.SMP.cached.thread-2] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Could not connect to: /10.99.248.138:5703. Reason: SocketException[Connection refused to address /10.99.248.138:5703] 2020-10-22 21:14:31,915 [hz.SMP.cached.thread-2] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [10.99.248.130]:5701 [dev] [3.7.5] [10.99.248.138]:5703 is added to the blacklist. 2020-10-22 21:14:31,915 [hz.SMP.cached.thread-2] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Connecting to /10.99.248.138:5701, timeout: 0, bind-any: true 2020-10-22 21:14:31,925 [hz.SMP.cached.thread-1] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Connecting to /10.99.248.130:5702, timeout: 0, bind-any: true 2020-10-22 21:14:31,933 [hz.SMP.cached.thread-1] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Could not connect to: /10.99.248.130:5702. Reason: SocketException[Connection refused to address /10.99.248.130:5702] 2020-10-22 21:14:31,933 [hz.SMP.cached.thread-1] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [10.99.248.130]:5701 [dev] [3.7.5] [10.99.248.130]:5702 is added to the blacklist. 2020-10-22 21:14:31,934 [hz.SMP.cached.thread-1] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Connecting to /10.99.248.130:5703, timeout: 0, bind-any: true 2020-10-22 21:14:31,934 [hz.SMP.cached.thread-1] INFO com.hazelcast.nio.tcp.InitConnectionTask - [10.99.248.130]:5701 [dev] [3.7.5] Could not connect to: /10.99.248.130:5703. Reason: SocketException[Connection refused to address /10.99.248.130:5703] 2020-10-22 21:14:31,934 [hz.SMP.cached.thread-1] INFO com.hazelcast.cluster.impl.TcpIpJoiner - [10.99.248.130]:5701 [dev] [3.7.5] [10.99.248.130]:5703 is added to the blacklist. 2020-10-22 21:14:31,947 [hz.SMP.cached.thread-2] INFO com.hazelcast.nio.tcp.TcpIpConnectionManager - [10.99.248.130]:5701 [dev] [3.7.5] Established socket connection between /10.99.248.130:41397 and /10.99.248.138:5701

1
What is your Hazelcast configuration?Rafał Leszko

1 Answers

0
votes

Seems like I've found the culprit. Our application uses a non-standard way to store Hazelcast's configuration. It uses a properties file stored as a BLOB inside a Database, its contents are:

HAZELCAST_MULTICAST_PORT= HAZELCAST_MULTICAST_TTL= HAZELCAST_MULTICAST_IP= HAZELCAST_MODE= HAZELCAST_UNICAST_IP=10.99.248.138,10.99.248.130 HAZELCAST_CONNECTION_TIMEOUT_SECONDS=

Removing 10.99.248.138 and restarting the application solved the mistery.