I have a small Hazelcast cluster running inside Openshift 4(kubernetes) in a number of pods. My problem is that when I roll out a new version, sometimes I need to change the configuration of the hazelcast instances in the new version. The problem then arises because the pods and the old cluster is still running, because the new pods aren't started yet properly yet. The new upgraded members will fail because their configurations are incompatible with the running cluster (The old pods). We are using the headless service discovery type described here. https://github.com/hazelcast/hazelcast-kubernetes. The data in the hazelcast cluster can be deleted between deployments, we have no need to keep anything from the old cluster when we start a new one.
I would like a way to tell the new members and the old members to either ignore their differences, start them up as two seperate clusters that don't find each other, or simply not replicate between the old and new members at all.
I have looked into the portable serialization feature https://docs.hazelcast.com/imdg/4.1.2/serialization/implementing-portable-serialization but it strikes me as overkill, as we don't need the cluster data to survive between deployments. I'm also unsure if it will help when we change something in the configuration for the hazelcast instances, which will make the clusters incompatible.
Lastly, I considered changing the deployment strategy for kubernetes so the old pods are killed before the new pods start up, but that will cause downtime for the application when we deploy, which is not great.
