redis-cluster on kubernetes: connection timed out

Question

I have combined/followed the following manuals to create a redis cluster on kubernetes (GCP):

I have created 3 nodes with each 2 pods on it. The problem is: I get a connection timeout when I connect from outside of the kubernetes cluster (through a load balancer external ip) to the redis-cluster.

$ redis-cli -h external_ip_lb -p 6379 -c

external_ip_lb:6379> set foo bar
-> Redirected to slot [12182] located at interal_ip_node:6379
Could not connect to Redis at interal_ip_node:6379: Operation timed out

When I get into the shell of a running container and do the redis-cli commands there, it works.

$ kubectl exec -it redis-cluster-0 -- redis-cli -c

127.0.0.1:6379> set foo bar
-> Redirected to slot [12182] located at internal_ip_node:6379
OK
internal_ip_node:6379> get foo
"bar"

I also tried to set a cluster IP service and do a port-foward to my local machine port 7000, this gives me the same error as with the external ip method.

$ kubectl port-foward pods/redis-cluster-0 7000:6379

Does anyone has an idea what could be wrong? Clearly it has something do do with my local machine not being a part of the kubernetes cluster, so the connection with the internal IP's of the other nodes fail.

Edit: output of kubectl describe svc redis-cluster-lb

Name:                     redis-cluster-lb
Namespace:                default
Labels:                   <none>
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"redis-cluster-lb","namespace":"default"},"spec":{"ports":[{"port"...
Selector:                 app=redis-cluster
Type:                     LoadBalancer
IP:                       internal_ip_lb
LoadBalancer Ingress:     external_ip_lb
Port:                     <unset>  6379/TCP
TargetPort:               6379/TCP
NodePort:                 <unset>  30631/TCP
Endpoints:                internal_ip_node_1:6379,internal_ip_node_2:6379,internal_ip_node_3:6379 + 3 more...
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

I'm able to ping the external load balancer's IP.

share output of kubectl describe svc servicename. are you able to ping the external_ip? — Arghya Sadhu

Matt Matt · Accepted Answer · 2020-05-25T07:50:18

I am not Redis expert, but in Redis documentation you can read:

Since cluster nodes are not able to proxy requests, clients may be redirected to other nodes using redirection errors

This is why you are are having this issues with redis cluster behind LB and this is also the reason why it is (most probably) not going to work.

You may probably need to use some proxy (e.g. official redis-cluster-poxy) that is running inside of k8s cluster, can reach all internal IPs of redis cluster and would handle redirects.

redis-cluster on kubernetes: connection timed out

1 Answers