Hazelcast cluster and Replication map synchronization

2
votes

I have a Hazelcast cluster with 2 instances (running from Docker containers) and a Replicated map which is filled at initialization at the first instance. All works fine and quick. Recently I experienced several times the following situation:

  1. first instance restarted all of sudden, joined the Hazelcast cluster and started to sync data but didn't finish
  2. second instance restarted immediately after as well for no reason; it joined the Hazelcast cluster and synced all data from first instance

I ended up with a healthy cluster but in reality the cluster contained only partial data which were synced before instance 2 shut down. It took at least one day to observe this bad state and to refresh the data.

This problem happened in multiple environments (test, prod). The reasons why the instances restarted are unknown. My Hazelcast version: 3.7.2. My assumption is the same can happen with 3 or more instances too with less probability.

What are the best practices in a case like this? Thanks!

2
javahazelcast
What means the instance restarted for no reason? When a JVM crashes it is normally not automatically restarted, Hazelcast itself doesn't restart anything. Not sure I get what you mean by restart? Do you run in a virtualized environment that might restart containers / VMs? - noctarius
I forgot to add (I edited my post, thanks). Restart = restart of docker container. - Theliro
Check the logs of the docker containers why they got restarted. I am suspecting they might be going OOM or something due to which the container might have got shutdown and container service started it again. Are you running the dockers on AWS ? - A.K.Desai

2 Answers

2
votes

if the second instance is restarted before the new instance got the state, you can not expect to recover the whole data.

0
votes

Cache is only caching data.Try to design your soft with a guess that cache may invalid or lose data all the time.There is also a machanism in hazelcast to load data from db when the data is not hitted.But this is not recommended,because it couples the cache and db.My proposal is that when data is lost, loading from db in your application.