Expose a Hazelcast cluster on AWS EKS with a load balancer

Question

We have a Hazelcast 3.12 cluster running inside an AWS EKS kubernetes cluster.

Do you know how to expose a Hazelcast cluster with more than 1 pod that is running inside an AWS EKS kubernetes cluster to outside the kubernetes cluster?

The Hazelcast cluster has 6 pods and is exposed outside of the kubernetes cluster with a kubernetes "Service" of type LoadBalancer (AWS classic load balancer).

When I run a Hazelcast client from outside of the kubernetes cluster, I am able to connect to the Hazelcast cluster using the AWS load balancer. However, when I try to get some value from a Hazelcast map, the client fails with this error:

java.io.IOException: No available connection to address [172.17.251.81]:5701 at com.hazelcast.client.spi.impl.SmartClientInvocationService.getOrTriggerConnect(SmartClientInvocationService.java:75

The error mentions the IP address 172.17.251.81. This is an internal kubernetes IP for a Hazelcast pod that I cannot connect to from outside the kubernetes cluster. I don't know why the client is trying to connect to this IP address instead of the Load Balancer public IP address.

On the other hand, when I scale the hazelcast cluster from 6 to 1 pod, I am able to connect and get the map value without any problem.

In case that you want to review the kubernetes LoadBalancer Service configuration:

kind: Service
apiVersion: v1
metadata:
  name: hazelcast-elb
  labels:
    app: hazelcast
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
spec:
  ports:
  - name: tcp-hazelcast-elb
    port: 443
    targetPort: 5701
  selector:
    app: hazelcast
  type: LoadBalancer

Rafał Leszko Rafał Leszko · Accepted Answer · 2020-09-09T08:08:41

If you expose all Pods with one LoadBalancer service, then you need to use Hazelcast Unisocket Client.

hazelcast-client:
  smart-routing: false

If you want to use the default Smart Client (which means better performance), then you need to expose each Pod with a separate service, because each Pod needs to be accessible from outside the Kubernetes cluster.

Read more in the blog post: How to Set Up Your Own On-Premises Hazelcast on Kubernetes.

Expose a Hazelcast cluster on AWS EKS with a load balancer

1 Answers