Automatic Hazelcast WAN Sync

Question

In the event of a cluster going down and then being restarted how can i maintain synchronously between cluster1 and cluster2?

They both have persistent data and active wan between each other but in the event that cluster2's wan queue is full while cluster1 is down, eventually when cluster1 goes back up the data centers would be out of sync.

Looking at the hazelcast documentation i see there is a way to initiate wan sync via man center as well as delta wan sync. Is there a way to initiate this on the startup of the cluster instead?

I assume i could have an event on cluster start which sends a rest call to man center to do this however i am wondering if its possible with rest calls and/or preferably without using man center at all.

Answer 1

WAN sync is not automatic, it can either be initiated through ManCenter or REST APIs. See links below: https://docs.hazelcast.org/docs/3.12.1/manual/html-single/index.html#synchronizing-wan-target-cluster

https://docs.hazelcast.org/docs/3.12.1/manual/html-single/index.html#rest-api

Important: since you are using WAN Replication, which is an enterprise feature, you are entitled to professional Hazelcast support and should be looking to communicate with them directly for SLA bound responses.

Answer 2

Adding a couple of comments to wildnez's answer:

Hazelcast has also implemented consistency checks between clusters using a Merkle Tree comparison and can synch only the delta's between the clusters rather than full cluster synch. Just something to keep in mind if using a hot restart or a persistence store that might load much of the cluster data at startup.