Hazelcast delay in multinode setup

Question

We are using Hazelcast in our project. We have a multi-node setup in which Hazelcast is running in all the three nodes (Hazelcast cluster). There is a latency delay in multi-node setup due to WAN latency (i.e) three nodes are in different locations.

If I ping from one node (say n1) to the other node (say n2), there is a consistent delay say 100ms.

But I am directly hitting to the one of the node (say like n1) from our web application, what we see is that there is a delay in the Hazelcast cast processing.

I am confused on the how Hazelcast works in a multi-node setup. Even if I hit directly to one of the nodes in the multi-node setup, why is there a delay in it?

I need the following information,

1) How does Hazelcast work in a multi-node cluster, and why there is a delay when I call any one of the nodes?

2) If so, how to do the tuning to avoid the delay?

3) Does Hazelcast contact to another node in case of get call?

Answer 1

Hazelcast partitions data, therefore each node holds a subset of the overall dataset. If you try to retrieve an element which is not hold locally (owned by the local node), you have to get it from a remote node. 100ms sounds like a pretty bad WAN connection and Hazelcast clusters are not designed to work over WAN.