Redis master becoming slave of another master in docker environment

Question

I have configured redis-sentinel with one master and two slaves.

lets call this setup of three machines a cluster.

I have a lot of clusters running on a lot of docker containers. On run time I manage the IP in the redis.conf file and sentinal.conf files.

My problem is; The master node on cluster-1 somehow became slave of the master of cluster-2. On Cluster-1 Master node I killed the redis and sentinel services, removed slaveof <cluster-2 master ip> 6379 and then restarted redis service with the edited conf file.

The moment I start redis service, It again becomes slave of cluster-2 master redis.

I tried slaveof no one from inside redis-cli but within seconds the node again turn into slave.

All this is happening without even starting sentinel service.

What is happening here? are there other entries that I would have to delete?

redis.conf

bind 0.0.0.0


protected-mode no

port 6379

tcp-backlog 511


timeout 0

tcp-keepalive 300


daemonize no

supervised no

pidfile "/var/run/redis_6379.pid"

loglevel notice

logfile "/var/log/redis.log"
databases 16


save 900 1
save 300 10
save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename "dump.rdb"

dir "/"



slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5



repl-disable-tcp-nodelay no
slave-priority 100                                                                                                    

appendonly no


appendfilename "appendonly.aof"


appendfsync everysec


no-appendfsync-on-rewrite no


auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

aof-load-truncated yes


lua-time-limit 5000
slowlog-log-slower-than 10000

slowlog-max-len 128


latency-monitor-threshold 0


notify-keyspace-events ""


hash-max-ziplist-entries 512
hash-max-ziplist-value 64

list-max-ziplist-size -2

list-compress-depth 0

set-max-intset-entries 512

zset-max-ziplist-entries 128
zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10

aof-rewrite-incremental-fsync yes
slaveof 192.168.60.38 6379 #this comes back again and again

sentinel config file may give you clues. As they might have details of existing master and slaves. — Nipun Talukdar
@NipunTalukdar I killed the sentinel service and deleted the file. All this happens without even starting sentinel. — Nagri

Nagri Nagri · Accepted Answer · 2018-02-20T12:03:51

For anyone who faces the same problem.

Sentinel is designed to automatically detect another sentinels in the same network.
So, Cluster-1 and Cluster-2 sentinels were all able to reach each other.

Sentinel of Cluster-1 became disloyal(pun intended) and rewrote the redis configuration of Cluster-1 redis making it slave of Cluster-2 Master redis.

Possible solutions;
1. Use Unique password between each redis setup. requirepass in redis configuration will be used.
2. Block traffic between different redis clusters.
3. Dont use sentinel all together.

Redis master becoming slave of another master in docker environment

1 Answers