Minimizing Hazelcast chatter and overhead

Question

Our app needs to run both on a large number of machines and on a single standalone machine. It has three distinct clusters, each performing a mostly isolated function. Cluster A is the main one, and clusters B & C are independent, but they both need access to a map in A to know where to route requests. Access needs to be super-fast.

Which setup should I choose?

1. Each cluster has its own Hazelcast instance. Clusters B & C are also lite members of the A instance.

2. Each cluster has its own Hazelcast instance. Clusters B & C use a Hazelcast client to talk to A.

3. One giant instance for all clusters.

I'm concerned about chatter and overhead as the clusters get larger, to potentially hundreds of machines. Which setup is most scalable?

Also, is there a writeup anywhere which details the messages that Hazelcast passes around? I'd like to know exactly what happens when a key gets added or removed, for example.

Answer 1

Try to avoid lite-member setup (1) as it is harder to maintain clusters with lite-members.

If all these machines/nodes are on the same local network and if # of nodes is around 50, you can go with (3).. all in one cluster. Otherwise I would go with (2) as you can scale clients really well and they are very lite-weight.