I have replicated your issue and found a solution to your problem.
Let me start by explaining what was the cause of your problem.
Redis gossip protocol works this way:
when you type cluster meet <ip> <port> on redis1,
redis1 opens a tcp connection to redis2.
In normal case, when redis2 receives a connection, it accepts it, looks up source ip address of who's connecting
and also opens a tcp connection to that address, so to redis1 in that case.
(More on how gossip protocol works inredis can be found in redis documentation,
or in this article)
Here comes the istio part.
Istio by default configures envoy as a typical proxy and as you can read in
istio documentation:
Istio be default is using REDIRECT proxing and as states in documentation:
This mode loses source IP addresses during redirection
This is our source of problem.
redis2 when receives a connection, it sees it as coming from localhost. Envoy has lost source IP address of redis1 and redis2 is now unable to
open a connection back to redis1.
Now, we have some options:
- you can try to change proxing mode to
TPROXY (I tried it but couldn't make it work)
- use redis built-in config variable
Lets take a look at the second option a bit closer because this is the one that worked for me.
In redis.conf
file you can find this section:
CLUSTER DOCKER/NAT support
In certain deployments, Redis Cluster nodes address discovery fails, because
addresses are NAT-ted or because ports are forwarded (the typical case is
Docker and other containers).
In order to make Redis Cluster working in such environments, a static
configuration where each node knows its public address is needed. The
following two options are used for this scope, and are:
- cluster-announce-ip
- cluster-announce-port
- cluster-announce-bus-port
Each instruct the node about its address, client port, and cluster message
bus port. The information is then published in the header of the bus packets
so that other nodes will be able to correctly map the address of the node
publishing the information.
If the above options are not used, the normal Redis Cluster auto-detection
will be used instead.
Note that when remapped, the bus port may not be at the fixed offset of
clients port + 10000, so you can specify any port and bus-port depending
on how they get remapped. If the bus-port is not set, a fixed offset of
10000 will be used as usually.
Example:
cluster-announce-ip 10.1.1.5
cluster-announce-port 6379
cluster-announce-bus-port 6380
We need to set cluster-announce-ip variable to redis's pod own ip address.
You can do it for example modifying redis-cluster ConfigMap like this
(It's modified redis configmap from this article):
---
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster
data:
update-node.sh: |
#!/bin/sh
REDIS_NODES="/data/nodes.conf"
sed -i -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/" ${REDIS_NODES}
cp /conf/redis.conf /redis.conf # <------HERE-----
sed -i "s/MY_IP/${POD_IP}/" /redis.conf # <------HERE-----
exec "$@"
redis.conf: |+
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
appendonly yes
protected-mode no
cluster-announce-ip MY_IP # <------HERE-----
Also remember to change you container's command like this to point to the right redis.conf file:
command: ["/conf/update-node.sh", "redis-server", "/redis.conf"]
Every redis node will now advertise this address as it's own so the other redis nodes will now know how to connect with it.
Let me know if it helped.
