Two separate hazelcast clusters in kubernetes

Question

For sharing events between the pods of two different services in a Kubernetes namespace, I intend to use Hazelcast. This is not a problem, however, each service also has a cluster that contains all its pods.

So, I have two clusters using the same pods. I achieved separation of the clusters by setting a group name for one of the clusters, while the other has the default group configuration. This works fine locally, with multiple instances of a test application. However, this is with multicast enabled.

In Kubernetes however, Hazelcast uses the HazelcastKubernetesDiscoveryStrategy and has Multicast disabled.

Both services have a label:

metadata:
  name: service-1
  labels:
    hazelcast-group: bc-events

metadata:
  name: service-2
  labels:
    hazelcast-group: bc-events

and the hazelcast configuration for the events cluster is like this:

Config hzConfig = new Config("events-instance");

NetworkConfig nwConfig = new NetworkConfig();
JoinConfig joinConfig = new JoinConfig();
joinConfig.setMulticastConfig(new MulticastConfig().setEnabled(false));
joinConfig.setTcpIpConfig(new TcpIpConfig().setEnabled(false));

DiscoveryStrategyConfig k8sDiscoveryStrategy = new DiscoveryStrategyConfig("com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy");
k8sDiscoveryStrategy.addProperty("namespace", "dev");
k8sDiscoveryStrategy.addProperty("resolve-not-ready-addresses", true);
k8sDiscoveryStrategy.addProperty("service-label-name", "hazelcast-group");
k8sDiscoveryStrategy.addProperty("service-label-value", "bc-events");

DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(k8sDiscoveryStrategy);
joinConfig.setDiscoveryConfig(discoveryConfig);
nwConfig.setJoin(joinConfig);

hzConfig.setNetworkConfig(nwConfig);
hzConfig.setProperty("hazelcast.discovery.enabled", "true");

GroupConfig groupConfig = new GroupConfig("bc-events");
hzConfig.setGroupConfig(groupConfig);

while the configuration for the shared cache cluster (the one without a group) is like this (for service 1, service 2 is the same):

Config hzConfig = new Config("service-1-app-hc");

NetworkConfig nwConfig = new NetworkConfig();
JoinConfig joinConfig = new JoinConfig();
joinConfig.setMulticastConfig(new MulticastConfig().setEnabled(false));
joinConfig.setTcpIpConfig(new TcpIpConfig().setEnabled(false));

DiscoveryStrategyConfig k8sDiscoveryStrategy = new DiscoveryStrategyConfig("com.hazelcast.kubernetes.HazelcastKubernetesDiscoveryStrategy");

k8sDiscoveryStrategy.addProperty("namespace", "dev");
k8sDiscoveryStrategy.addProperty("service-name", "service-1");
k8sDiscoveryStrategy.addProperty("resolve-not-ready-addresses", true);

DiscoveryConfig discoveryConfig = new DiscoveryConfig();
discoveryConfig.addDiscoveryStrategyConfig(k8sDiscoveryStrategy);
joinConfig.setDiscoveryConfig(discoveryConfig);
nwConfig.setJoin(joinConfig);

hzConfig.setNetworkConfig(nwConfig);
hzConfig.setProperty("hazelcast.discovery.enabled", "true");

The hazelcast instances find eachother, but then complain that the other has a different group name and blacklists the IP.

While debugging the code, during config validation while processing a join request, it tries to compare the group names (bc-events against dev) and obviously that's different. But then this gets blacklisted (I believe), preventing a validation check for the other instance that has the same group name.

I'm not sure where to go next. I cannot test this configuration locally because without multicast, it doesn't find the other nodes for joining the cluster. I also don't think there is anything wrong with the configuration.

The libraries used are:

<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast</artifactId>
    <version>3.7.8</version>
</dependency>
<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-kubernetes</artifactId>
    <version>1.1.0</version>
</dependency>

UPDATE:

I should note that, the current setup, where this cluster that has the group name discovers by service name (as such, it only contains the pods of one service) actually works. The cluster without a group and the cluster with a group are running alongside eachother. It is only when I switch to label-based discovery (and that the other service gets involved) that it breaks.

UPDATE:

When changing the port of the events cluster, I noticed it still tries to connect to 5701 (the default) despite being put on 5801. Of course this works because the first cluster is running on 5701. Inside HazelcastKubernetesDiscoveryStrategy there is the following method:

protected int getServicePort(Map<String, Object> properties) {
    int port = NetworkConfig.DEFAULT_PORT;
    if (properties != null) {
        String servicePort = (String) properties.get(HAZELCAST_SERVICE_PORT);
        if (servicePort != null) {
            port = Integer.parseInt(servicePort);
        }
    }
    return port;
}

This method checks the additional properties that get returned for each endpoint returned by the kubernetes client, for a hazelcast port configuration. If none exists, it uses the default 5701. I am guessing this is what needs to be configured, however, it must not impact the other cluster, so I may have to extend the strategy with some of my own logic.

Answer 1

It's possible to embed multiple Hazelcast instances into an application deployed in a single POD. Then, you can control how the cluster are formed. It requires an additional configuration, but you don't have to modify HazelcastKubernetesDiscoveryStrategy.

Sample application

I've created a sample application to present how it works. Please check it here: https://github.com/leszko/hazelcast-code-samples/tree/kubernetes-embedded-multiple/hazelcast-integration/kubernetes/samples/embedded.

Configuration steps

Hazelcast Configuration

You have two Hazelcast instances in your application, so you need to specify the ports they use. Also, with the parameters of the hazelcast-kubernetes plugin, you can configure which Hazelcast instances form the cluster together.

For example, assuming the first Hazelcast instance should form a cluster with all other Hazelcast instances in the current Kubernetes namespace, its configuration can look as follows.

Config config = new Config();
config.getNetworkConfig().setPort(5701);
config.getProperties().setProperty("hazelcast.discovery.enabled", "true");
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);

HazelcastKubernetesDiscoveryStrategyFactory discoveryStrategyFactory = new HazelcastKubernetesDiscoveryStrategyFactory();
Map<String, Comparable> properties = new HashMap<>();
joinConfig.getDiscoveryConfig().addDiscoveryStrategyConfig(new DiscoveryStrategyConfig(discoveryStrategyFactory, properties));

Then, the second Hazelcast instance could form the cluster only with the same application. We could make this separation by giving it a service-name parameter with the value from environment variables.

Config config = new Config();
config.getNetworkConfig().setPort(5702);
config.getProperties().setProperty("hazelcast.discovery.enabled", "true");
JoinConfig joinConfig = config.getNetworkConfig().getJoin();
joinConfig.getMulticastConfig().setEnabled(false);

HazelcastKubernetesDiscoveryStrategyFactory discoveryStrategyFactory = new HazelcastKubernetesDiscoveryStrategyFactory();
Map<String, Comparable> properties = new HashMap<>();
String serviceName = System.getenv("KUBERNETES_SERVICE_NAME");
properties.put("service-name", serviceName);
properties.put("service-port", "5702");
joinConfig.getDiscoveryConfig().addDiscoveryStrategyConfig(new DiscoveryStrategyConfig(discoveryStrategyFactory, properties));

GroupConfig groupConfig = new GroupConfig("separate");

Kubernetes Template

Then, in the Kubernetes Deployment template, you need to configure both ports: 5701 and 5702.

ports:
- containerPort: 5701
- containerPort: 5702

And the environment variable with the service name:

env:
- name: KUBERNETES_SERVICE_NAME
  value: hazelcast-separate-1

Plus, you need to create two services, for each Hazelcast instance.

kind: Service
metadata:
  name: hazelcast-shared-1
spec:
  type: ClusterIP
  selector:
    app: hazelcast-app
  ports:
  - name: hazelcast-shared
    port: 5701

kind: Service
metadata:
  name: hazelcast-separate-1
spec:
  type: ClusterIP
  selector:
    app: hazelcast-app
  ports:
  - name: hazelcast-separate
  port: 5702

Obviously, in the same manner, you can separate Hazelcast clusters using service labels instead of service names.