redis 2.8.7 sentinel environment configuration questions for linux, how to make it autostart, what they should subscribe to?

2
votes

for now we're trying to play with redis 2.8.7 as cache storage (from the .NET web application using booksleeve client). It seems to be very interesting and exciting task at the moment, redis documentation is very nice, but due to lack of real practical experience I do have couple of questions about how expected configuration should be done properly.

I took next articles as main configuration source:

  1. Installing redis with autostart capability (using an init script, so that after a restart everything will start again properly) : http://redis.io/topics/quickstart
  2. Deployment of the redis into azure: http://haishibai.blogspot.com/2014/01/walkthrough-setting-up-redis-cluster-on.html

Initial idea/assumption - is to have 1 redis master and 2 slave instances running with linux Ubuntu. In order to provide high availability of the instances - I've decided to use sentinel. So my expected configuration looks like this at the moment:

  1. MasterInstance: VM1 (linux, Ubuntu), port : 6379 (autostarted when linux is restarted)
  2. Slave1: VM2 (linux, ubuntu), port : 6380 (autostarted when linux is restarted) : slaveOf MasterID 6379
  3. Slave2: VM3 (linux, ubuntu), port : 6379 (autostarted when linux is restarted) : slaveOf MasterIP 6379

After VMs started, I can see that I've got 2 slaves successfully connected and syncing with Master: Trace sample from the master:

[1120] 25 Mar 14:11:18.629 - 1 clients connected (0 slaves), 793352 bytes in use
[1120] 25 Mar 14:11:18.634 * Slave asks for synchronization
[1120] 25 Mar 14:11:18.634 * Full resync requested by slave.
[1120] 25 Mar 14:11:18.634 * Starting BGSAVE for SYNC
[1120] 25 Mar 14:11:18.634 * Background saving started by pid 1227
[1227] 25 Mar 14:11:18.810 * DB saved on disk
[1227] 25 Mar 14:11:18.810 * RDB: 0 MB of memory used by copy-on-write
[1120] 25 Mar 14:11:18.836 * Background saving terminated with success
[1120] 25 Mar 14:11:18.837 * Synchronization with slave succeeded
[1120] 25 Mar 14:11:23.829 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:23.829 - DB 2: 4 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:23.829 - 0 clients connected (1 slaves), 1841992 bytes in use
[1120] 25 Mar 14:11:29.011 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:29.011 - DB 2: 4 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:29.011 - 0 clients connected (1 slaves), 1841992 bytes in use
[1120] 25 Mar 14:11:29.826 - Accepted 168.62.36.189:1024
[1120] 25 Mar 14:11:29.828 * Slave asks for synchronization
[1120] 25 Mar 14:11:29.828 * Full resync requested by slave.
[1120] 25 Mar 14:11:29.828 * Starting BGSAVE for SYNC
[1120] 25 Mar 14:11:29.828 * Background saving started by pid 1321
[1321] 25 Mar 14:11:29.871 * DB saved on disk
[1321] 25 Mar 14:11:29.871 * RDB: 0 MB of memory used by copy-on-write
[1120] 25 Mar 14:11:29.943 * Background saving terminated with success
[1120] 25 Mar 14:11:29.946 * Synchronization with slave succeeded
[1120] 25 Mar 14:11:34.195 - DB 0: 2 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:34.195 - DB 2: 4 keys (0 volatile) in 4 slots HT.
[1120] 25 Mar 14:11:34.195 - 0 clients connected (2 slaves), 1862920 bytes in use

now I need to setup sentinel instances ...

  1. I copied sentinel.conf from the initial redis-stable package into 3 VM runnung redis (1 master and both slaves)

  2. Inside each config I've done next modifications:

    sentinel monitor mymaster MasterPublicIP 6379 2

  3. on each VM started sentinel using next command line:

    redis-server /etc/redis/sentinel.conf -- sentinel

After that I've got the response that sentinel successfully started ... on all VMs... After I started all 3 sentinel instances I've got next trace sample (sentinel.conf files were updated with information about slaves and other sentinel instances):

[1743] 25 Mar 16:35:46.450 # Sentinel runid is 05380d689af9cca1e826ce9c85c2d68c65780878
[1743] 25 Mar 16:35:46.450 # +monitor master mymaster MasterIP 6379 quorum 2
[1743] 25 Mar 16:36:11.578 * -dup-sentinel master mymaster MasterIP 6379 #duplicate of     10.119.112.41:26379 or 83666bdd03fd064bcf2ec41ec2134d4e1e239842
[1743] 25 Mar 16:36:11.578 * +sentinel sentinel 10.119.112.41:26379 10.119.112.41 26379 @ mymaster 168.62.41.1 6379
[1743] 25 Mar 16:36:16.468 # +sdown sentinel 10.175.220.134:26379 10.175.220.134 26379 @ mymaster 168.62.41.1 6379
[1743] 25 Mar 16:36:40.876 * -dup-sentinel master mymaster MasterIP 6379 #duplicate of 10.175.220.134:26379 or fe9edeb321e04070c6ac6e28f52c05317a593ffd
[1743] 25 Mar 16:36:40.876 * +sentinel sentinel 10.175.220.134:26379 10.175.220.134 26379 @ mymaster 168.62.41.1 6379
[1743] 25 Mar 16:37:10.962 # +sdown sentinel 10.175.220.134:26379 10.175.220.134 26379 @ mymaster 168.62.41.1 6379

based on the trace sample, I have next questions. It will be great, if someone can clarify them:

  1. Why do I see -dup-sentinel master mymaster configuration here ... Is it because I added 3 sentinels for the same master instance (maybe I need to register 1 sentinel per instance of redis - so 1 sentinel is going to be mapped to the master and 2 other sentinels - to the 2 slaves)?
  2. how to start sentinels in the way redis servers is started (automatically even then VM is restarted)? - do I need to perform same actions and register them as ordinary redis-server instances?
  3. Is it ok to have sentinel instance to be hosted in the same VM as redis-server?

After that I started new putty connection and started redis-cli to work with sentinel APIs, but received next response on my command below: 

127.0.0.1:6379> SENTINEL masters

(error) ERR unknown command 'SENTINEL'

I guess I've done something stupid here... :( What I've done wrong and how to test sentinel APIs from the terminal connection?

Thank you in advance for any help.

2
redissentinel

2 Answers

14
votes

I guess "SENTINEL masters" should be run on the Redis sentinel

redis-cli -p 26379 (which the default sentinel port)

then issue

127.0.0.1:26379> SENTINEL masters

and you will get something

1) "name" 2) "mymaster" 3) "ip" 4) "127.0.0.1" 5) "port" 6) "6379" . . .

To start sentinels automatically even then VM is restarted

first set daemonize yes into sentinel.conf

and modify the init script here (https://github.com/antirez/redis/blob/unstable/utils/redis_init_script) to reflect the sentinel port and .conf location.

$EXEC $CONF --sentinel # starting in Sentinel mode

and the rest is like you did for redis server.

1
votes

First, you don't run Sentinel on the master. Sentinel is designed to detect when the master fails. If you run Sentinel on the same system as the master, you will lose a Sentinel when you lose the system. For the same reasons you shouldn't use the slaves as your additional test points.

You want to run Sentinel from where the clients run - to ensure you are testing for network outages.

Next, you mention you added slave information to your sentinel configs. You don't configure slaves in sentinel - it discovers them through the master. I suspect you've added additional sentinel monitor commands for each slave - this would indeed cause duplicate monitoring attempts.

Third, as @yofpro mentioned, to run sentinel commands you need to connect to sentinel -not Redis master or slaves.