In my project I have 2 servers running an application. They run two Hazelcast nodes which are initialized as Spring beans. I use Hibernate and its second level cache is also supported by Hazelcast. There is one distributed map, that is backed by a maploader that primes the cache in the beginning. If one of the 2 servers is restarted (node B), the following situation happens:
- node A sees that node B is gone and periodically tries to reconnect
- node B starts initializing the Spring context, during this the Hazelcast instance also starts
- node A joins node B and some migration (?) operations hit node B, which tries to get the above mentioned map
- in parallel Spring initializes the JPA entity manager which also tries to access the map (for setting up L2 cache)
- this causes a deadlock: Spring init thread holds a lock related to the bean map and tries to get the distributed map, while the migration operations hold the map and try to get the maploader bean from Spring
My problem is that I don't have control over when Hazelcast first accepts connection from its peers.
My HZ bean is defined like this:
@Bean
public void hazelcast() {
Hazelcast.newInstance(config());
}
and I can't wait with accepting connections until Spring is fully booted. I could not find any way to prevent Hazelcast from doing operations before my app is fully up.
How could I start the Hazelcast TCP/IP network interface programmatically? Or how can I deny connections? Can you suggest a deadlock-free pattern to achieve being L2 cache for JPA, doing a cache prime on startup with a maploader bean that also uses JPA to load a particular distributed map?
