I am trying to setup a redis replication cluster with 3 redis servers (1 primary and 2 replicas) and 3 redis sentinels. One server and database pair will exist on a machine and there will be 3 machines, each machine has docker installed. The issue I am having is that the redis server instances are unable to connect to MASTER and resolve the primary name. There seems to be no issues with the network or port openings since I am able to specify each machines IPv4 in the configuration below and the communication works as expected.
Redis version available in logs further down
docker-compose version 1.29.2, build 5becea4c
Docker version 20.10.14, build a224086
Ubuntu 20.04.4 LTS
Below are the docker-compose.yml files:
I did not include redis_3 config and output since it is almost identical to redis_2
primary:
version: '3.8'
services:
redis_1:
image: bitnami/redis:6.2
restart: always
command: ["redis-server", "--protected-mode", "no", "--dir", "/data"]
environment:
- REDIS_REPLICA_IP=redis_1
- REDIS_REPLICATION_MODE=master
- REDIS_MASTER_PASSWORD=very-good-password
- REDIS_PASSWORD=very-good-password
ports:
- "6379:6379"
volumes:
- "/opt/knowit/docker/data:/data"
sentinel_1:
image: bitnami/redis-sentinel:6.2
restart: always
environment:
- REDIS_MASTER_HOST=redis_1
- REDIS_MASTER_PASSWORD=very-good-password
- REDIS_SENTINEL_ANNOUNCE_IP=redis_1
- REDIS_SENTINEL_QUORUM=2
- REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=5000
- REDIS_SENTINEL_FAILOVER_TIMEOUT=60000
- REDIS_SENTINEL_PASSWORD=other-good-password
- REDIS_SENTINEL_ANNOUNCE_HOSTNAMES=yes
- REDIS_SENTINEL_RESOLVE_HOSTNAMES=yes
ports:
- "26379:26379"
replica:
version: '3.8'
services:
redis_2:
image: bitnami/redis:6.2
restart: always
command: ["redis-server", "--protected-mode", "no", "--replicaof", "redis_1", "6379", "--dir", "/data"]
environment:
- REDIS_REPLICA_IP=redis_2
- REDIS_REPLICATION_MODE=replica
- REDIS_MASTER_PASSWORD=very-good-password
- REDIS_PASSWORD=very-good-password
ports:
- "6379:6379"
volumes:
- "/opt/knowit/docker/data:/data"
sentinel_2:
image: bitnami/redis-sentinel:6.2
restart: always
environment:
- REDIS_MASTER_HOST=redis_1
- REDIS_MASTER_PASSWORD=very-good-password
- REDIS_SENTINEL_ANNOUNCE_IP=redis_2
- REDIS_SENTINEL_QUORUM=2
- REDIS_SENTINEL_DOWN_AFTER_MILLISECONDS=5000
- REDIS_SENTINEL_FAILOVER_TIMEOUT=60000
- REDIS_SENTINEL_PASSWORD=other-good-password
- REDIS_SENTINEL_ANNOUNCE_HOSTNAMES=yes
- REDIS_SENTINEL_RESOLVE_HOSTNAMES=yes
ports:
- "26379:26379"
The docker logs looks like this:
primary:
$ sudo docker-compose up
Starting docker_redis_1 ... done
Starting docker_sentinel_1 ... done
Attaching to docker_redis_1, docker_sentinel_1
redis_1 | redis 12:15:41.03
redis_1 | redis 12:15:41.04 Welcome to the Bitnami redis container
redis_1 | redis 12:15:41.04 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
redis_1 | redis 12:15:41.04 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
redis_1 |
redis_1 | redis 12:15:41.04
redis_1 | 1:C 29 Apr 2022 12:15:41.068 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 29 Apr 2022 12:15:41.069 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 29 Apr 2022 12:15:41.069 # Configuration loaded
redis_1 | 1:M 29 Apr 2022 12:15:41.070 * monotonic clock: POSIX clock_gettime
redis_1 | 1:M 29 Apr 2022 12:15:41.072 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
redis_1 | 1:M 29 Apr 2022 12:15:41.072 * Running mode=standalone, port=6379.
redis_1 | 1:M 29 Apr 2022 12:15:41.072 # Server initialized
redis_1 | 1:M 29 Apr 2022 12:15:41.073 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 29 Apr 2022 12:15:41.073 * Ready to accept connections
sentinel_1 | redis-sentinel 12:15:41.08
sentinel_1 | redis-sentinel 12:15:41.08 Welcome to the Bitnami redis-sentinel container
sentinel_1 | redis-sentinel 12:15:41.08 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis-sentinel
sentinel_1 | redis-sentinel 12:15:41.08 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis-sentinel/issues
sentinel_1 | redis-sentinel 12:15:41.08
sentinel_1 | redis-sentinel 12:15:41.08 INFO ==> ** Starting Redis sentinel setup **
sentinel_1 | redis-sentinel 12:15:41.11 INFO ==> Initializing Redis Sentinel...
sentinel_1 | redis-sentinel 12:15:41.11 INFO ==> Persisted files detected, restoring...
sentinel_1 | redis-sentinel 12:15:41.12 INFO ==> ** Redis sentinel setup finished! **
sentinel_1 |
sentinel_1 | redis-sentinel 12:15:41.13 INFO ==> ** Starting Redis Sentinel **
sentinel_1 | 1:X 29 Apr 2022 12:15:41.143 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
sentinel_1 | 1:X 29 Apr 2022 12:15:41.144 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
sentinel_1 | 1:X 29 Apr 2022 12:15:41.144 # Configuration loaded
sentinel_1 | 1:X 29 Apr 2022 12:15:41.145 * monotonic clock: POSIX clock_gettime
sentinel_1 | 1:X 29 Apr 2022 12:15:41.146 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
sentinel_1 | 1:X 29 Apr 2022 12:15:41.147 * Running mode=sentinel, port=26379.
sentinel_1 | 1:X 29 Apr 2022 12:15:41.148 # Sentinel ID is 232f6b838b76c348f123597f2852091a77bdae03
sentinel_1 | 1:X 29 Apr 2022 12:15:41.148 # +monitor master mymaster redis_1 6379 quorum 2
replica:
$ sudo docker-compose up
Starting docker_redis_2 ... done
Starting docker_sentinel_2 ... done
Attaching to docker_redis_2, docker_sentinel_2
redis_2 | redis 11:53:24.61
redis_2 | redis 11:53:24.62 Welcome to the Bitnami redis container
redis_2 | redis 11:53:24.63 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis
redis_2 | redis 11:53:24.63 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis/issues
redis_2 | redis 11:53:24.63
redis_2 |
redis_2 | 1:C 29 Apr 2022 11:53:24.649 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_2 | 1:C 29 Apr 2022 11:53:24.651 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
redis_2 | 1:C 29 Apr 2022 11:53:24.651 # Configuration loaded
redis_2 | 1:S 29 Apr 2022 11:53:24.653 * monotonic clock: POSIX clock_gettime
redis_2 | 1:S 29 Apr 2022 11:53:24.656 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
redis_2 | 1:S 29 Apr 2022 11:53:24.657 * Running mode=standalone, port=6379.
redis_2 | 1:S 29 Apr 2022 11:53:24.657 # Server initialized
redis_2 | 1:S 29 Apr 2022 11:53:24.657 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_2 | 1:S 29 Apr 2022 11:53:24.659 * Ready to accept connections
redis_2 | 1:S 29 Apr 2022 11:53:24.659 * Connecting to MASTER redis_1:6379
sentinel_2 | redis-sentinel 11:53:24.70
sentinel_2 | redis-sentinel 11:53:24.71 Welcome to the Bitnami redis-sentinel container
sentinel_2 | redis-sentinel 11:53:24.71 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis-sentinel
sentinel_2 | redis-sentinel 11:53:24.71 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis-sentinel/issues
sentinel_2 | redis-sentinel 11:53:24.71
sentinel_2 | redis-sentinel 11:53:24.71 INFO ==> ** Starting Redis sentinel setup **
redis_2 | 1:S 29 Apr 2022 11:53:34.673 # Unable to connect to MASTER: Resource temporarily unavailable
sentinel_2 | redis-sentinel 11:53:34.75 WARN ==> Hostname redis_1 could not be resolved, this could lead to connection issues
sentinel_2 | redis-sentinel 11:53:34.76 INFO ==> Initializing Redis Sentinel...
sentinel_2 | redis-sentinel 11:53:34.76 INFO ==> Persisted files detected, restoring...
sentinel_2 | redis-sentinel 11:53:34.77 INFO ==> ** Redis sentinel setup finished! **
sentinel_2 |
sentinel_2 | redis-sentinel 11:53:34.79 INFO ==> ** Starting Redis Sentinel **
redis_2 | 1:S 29 Apr 2022 11:53:35.675 * Connecting to MASTER redis_1:6379
sentinel_2 | 1:X 29 Apr 2022 11:53:44.813 # Failed to resolve hostname 'redis_1'
sentinel_2 | 1:X 29 Apr 2022 11:53:44.813 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
sentinel_2 | 1:X 29 Apr 2022 11:53:44.813 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=1, just started
sentinel_2 | 1:X 29 Apr 2022 11:53:44.813 # Configuration loaded
sentinel_2 | 1:X 29 Apr 2022 11:53:44.814 * monotonic clock: POSIX clock_gettime
sentinel_2 | 1:X 29 Apr 2022 11:53:44.815 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
sentinel_2 | 1:X 29 Apr 2022 11:53:44.815 * Running mode=sentinel, port=26379.
sentinel_2 | 1:X 29 Apr 2022 11:53:44.816 # Sentinel ID is bfec501e81d8da33def75f23911b606aa395078d
sentinel_2 | 1:X 29 Apr 2022 11:53:44.816 # +monitor master mymaster redis_1 6379 quorum 2
sentinel_2 | 1:X 29 Apr 2022 11:53:44.817 # +tilt #tilt mode entered
redis_2 | 1:S 29 Apr 2022 11:53:45.687 # Unable to connect to MASTER: Resource temporarily unavailable
redis_2 | 1:S 29 Apr 2022 11:53:46.689 * Connecting to MASTER redis_1:6379
sentinel_2 | 1:X 29 Apr 2022 11:53:54.831 # Failed to resolve hostname 'redis_1'
sentinel_2 | 1:X 29 Apr 2022 11:53:54.914 # +tilt #tilt mode entered
redis_2 | 1:S 29 Apr 2022 11:53:56.701 # Unable to connect to MASTER: Resource temporarily unavailable
Is there some issue with resolving hostnames when the different redis instances are located on separate machines or have I just missed something basic?
I would assume the latter, since I have been able to get this up and running by specifying the ip addresses and also the replica receives the hostname of the primary.
Any help would be much appreciated! Let me know if additional information is required
