Camel cluster on kubernetes using Hazelcast error when rolling updates

Question

I have an embedded Hazelcast instance that I configure to use k8s API (service discovery) and I use Route Policy to have an Active/Passive clustered FTP reader.

I have min of 3 instances in the cluster. My rolling updates are first adding a new instance before replacing old ones.

I am getting this error when I do rolling updates.

[com.hazelcast.core.HazelcastInstanceNotActiveException - Hazelcast instance is not active!]com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active! 
at com.hazelcast.spi.impl.AbstractDistributedObject.throwNotActiveException(AbstractDistributedObject.java:115) 
at com.hazelcast.spi.impl.AbstractDistributedObject.lifecycleCheck(AbstractDistributedObject.java:110) 
at com.hazelcast.spi.impl.AbstractDistributedObject.getNodeEngine(AbstractDistributedObject.java:104) 
at com.hazelcast.map.impl.proxy.MapProxyImpl.tryLock(MapProxyImpl.java:500) 
at org.apache.camel.component.hazelcast.policy.HazelcastRoutePolicy.acquireLeadership(HazelcastRoutePolicy.java:296) 
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
at java.base/java.lang.Thread.run(Thread.java:834)

It looks like the Hazelcast instance is stopping before the camel has a chance to elect a new leader for this cluster.

Questions:

Do I need to add the instance to the camel component when I create a Hazelcast instance in code?
How do I control the shutdown order to make sure the lock is migrated to a different node?
Any code examples for how this should be done?

I am using the Hazelcast kubernetes plugin

    Config config = new Config();

    config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);

    config
        .getNetworkConfig()
        .getJoin()
        .getKubernetesConfig()
        .setEnabled(true)
        .setProperty("namespace", "aaaa")
        .setProperty("service-name", "bbbb");

    HazelcastInstance hazelcastInstance = Hazelcast.newHazelcastInstance(config);

I then create a route policy

    HazelcastRoutePolicy routePolicy = new HazelcastRoutePolicy(hz);
    routePolicy.setLockMapName("etl");
    routePolicy.setLockKey("etl-key");
    routePolicy.setLockValue("etl-value");
    routePolicy.setTryLockTimeout(5, TimeUnit.SECONDS);
    context.getRegistry().bind("cluster", routePolicy);

and use this policy in my route:

        .routePolicyRef("cluster")

Thanks in advance :)

EDIT

I think this is related, I always have double the number of Hazelcast nodes as Pods. I start a 3 containers cluster and see 6 in logs.

Members {size:6, ver:6} [
      Member [172.30.0.3]:5701 - ae3861bb-3f77-427e-ada1-5b669c89776f
      Member [172.30.0.4]:5701 - cd7741f0-a3b1-4883-a8bb-c3fecaad6b14 this
      Member [172.30.0.2]:5701 - 3403bbee-8141-4572-bbd2-4379bf452e79
      Member [172.30.0.3]:5702 - 0d48a691-b166-4d29-9b34-77b93be4370b
      Member [172.30.0.4]:5702 - d36f0342-730e-40e6-a18e-cf72d9d4b574
      Member [172.30.0.2]:5702 - f3644c6c-afe7-4f48-9832-b604830b6955

Rafał Leszko Rafał Leszko · Accepted Answer · 2021-02-15T15:44:39

If you use Hazelcast with Rolling Upgrade (or scaling down), you should configure graceful shutdown as described here:

Camel cluster on kubernetes using Hazelcast error when rolling updates

1 Answers