I am trying to test redis sentinel cluster failover, implemented with a docker-compose file and the communication is hostname based. To simulate failover I stop the current redis master container example docker stop redis1, what I expect is sentinel failover and one of the replicated redis instances is promoted as a new master, but what I get is the log below.
edis_sentinel-redis1-1 | 1:signal-handler (1648684795) Received SIGTERM scheduling shutdown...
redis_sentinel-redis1-1 | 1:M 30 Mar 2022 23:59:55.424 # User requested shutdown...
redis_sentinel-redis1-1 | 1:M 30 Mar 2022 23:59:55.424 * Saving the final RDB snapshot before exiting.
redis_sentinel-redis1-1 | 1:M 30 Mar 2022 23:59:55.430 * DB saved on disk
redis_sentinel-redis1-1 | 1:M 30 Mar 2022 23:59:55.430 # Redis is now ready to exit, bye bye...
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.432 # Connection with master lost.
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.432 * Caching the disconnected master state.
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:55.432 # Connection with master lost.
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.433 * Reconnecting to MASTER redis1:6379
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:55.432 * Caching the disconnected master state.
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:55.433 * Reconnecting to MASTER redis1:6379
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:55.434 * MASTER <-> REPLICA sync started
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:55.434 # Error condition on socket for SYNC: Connection refused
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.434 * MASTER <-> REPLICA sync started
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.434 # Error condition on socket for SYNC: Connection refused
redis_sentinel-redis1-1 exited with code 0
redis_sentinel-redis1-1 exited with code 0
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.724 * Connecting to MASTER redis1:6379
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:55.813 # Unable to connect to MASTER: Invalid argument
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:56.129 * Connecting to MASTER redis1:6379
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:56.170 # Unable to connect to MASTER: Invalid argument
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:56.825 * Connecting to MASTER redis1:6379
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:56.870 # Unable to connect to MASTER: Invalid argument
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:57.176 * Connecting to MASTER redis1:6379
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:57.224 # Unable to connect to MASTER: Invalid argument
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:57.878 * Connecting to MASTER redis1:6379
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:57.929 # Unable to connect to MASTER: Invalid argument
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:58.231 * Connecting to MASTER redis1:6379
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:58.305 # Unable to connect to MASTER: Invalid argument
redis_sentinel-sentinel1-1 | 1:X 30 Mar 2022 23:59:58.461 # +sdown master redismaster redis1 6379
redis_sentinel-sentinel2-1 | 1:X 30 Mar 2022 23:59:58.506 # +sdown master redismaster redis1 6379
redis_sentinel-sentinel3-1 | 1:X 30 Mar 2022 23:59:58.550 # +sdown master redismaster redis1 6379
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:58.935 * Connecting to MASTER redis1:6379
redis_sentinel-redis3-1 | 1:S 30 Mar 2022 23:59:58.977 # Unable to connect to MASTER: Invalid argument
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:59.311 * Connecting to MASTER redis1:6379
redis_sentinel-redis2-1 | 1:S 30 Mar 2022 23:59:59.352 # Unable to connect to MASTER: Invalid argument
To Reproduce:
Files structure
|./docker-compose.yaml
|./sentinel1/sentinel.conf
|./sentinel2/sentinel.conf
|./sentinel3/sentinel.conf
docker-compose.yaml
version: '3.8'
x-base: &base
image: redis:6.2.6
x-base-sen: &base-sen
image: redis:6.2.6
command: redis-sentinel /sentinel/sentinel.conf
services:
redis1:
<<: *base
hostname: redis1
command: redis-server --replica-announce-ip redis1
redis2:
<<: *base
hostname: redis2
command: redis-server --slaveof redis1 6379 --replica-announce-ip redis2
redis3:
<<: *base
hostname: redis3
command: redis-server --slaveof redis1 6379 --replica-announce-ip redis3
sentinel1:
<<: *base-sen
hostname: sentinel1
volumes:
- type: bind
source: sentinel1
target: /sentinel
sentinel2:
<<: *base-sen
hostname: sentinel2
volumes:
- type: bind
source: sentinel2
target: /sentinel
sentinel3:
<<: *base-sen
hostname: sentinel3
volumes:
- type: bind
source: sentinel3
target: /sentinel
sentinel1/sentinel.conf
port 26379
dir /tmp
sentinel monitor redismaster redis1 6379 2
sentinel down-after-milliseconds redismaster 3000
sentinel parallel-syncs redismaster 1
sentinel failover-timeout redismaster 6000
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel announce-ip sentinel1
sentinel2/sentinel.conf
port 26379
dir /tmp
sentinel monitor redismaster redis1 6379 2
sentinel down-after-milliseconds redismaster 3000
sentinel parallel-syncs redismaster 1
sentinel failover-timeout redismaster 6000
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel announce-ip sentinel2
sentinel3/sentinel.conf
port 26379
dir /tmp
sentinel monitor redismaster redis1 6379 2
sentinel down-after-milliseconds redismaster 3000
sentinel parallel-syncs redismaster 1
sentinel failover-timeout redismaster 6000
sentinel resolve-hostnames yes
sentinel announce-hostnames yes
sentinel announce-ip sentinel3
