How to switch masters in this Redis Sentinel configuration?

1
votes

I have the following Redis/Sentinel configuration:

  • Redis master A + N slaves
  • M sentinels watching A, named masterA
  • the client application query the sentinels for masterA, then query and modify A

Now say A is outdated and I want to replace it by a new Redis master called B (with minimum down time / data loss.). In the end of the operation, I want this:

  • Redis master B + N slaves
  • the client application querying and modifying B

I could proceed as follows:

  1. Have the sentinels start watching B, named masterB
  2. Have each slave of A become a slave of B

From there, I am stuck because the client application still asks for masterA when talking to the sentinels. I have two questions:

  • Is there a way to switch masters names, such that B becomes known as masterA for the sentinels, and therefore for the client application as well?
  • Is it better to modify the client application code to handle the switch from an old master to a new master?
1
redisredis-sentinel

1 Answers

1
votes

One way of achieving your aim is to follow the age old solution of "adding another level of indirection".

A particularly effective method is to have your clients talk to a TCP proxy (e.g. HAProxy) and have it pass the traffic to the current master.

To keep the TCP proxy is sync you can do something similar to http://blog.haproxy.com/2014/01/02/haproxy-advanced-redis-health-check/ which makes HAProxy Sentinel aware.

The major plus for this solution is that it makes your clients very simple - they only connect to one place and the traffic is always forwarded to the correct Redis instance.

One issue with this solution is that HAProxy's configuration DSL does not have the ability to deal with the period when a Redis server restarts and announces itself initially as a master before the sentinels make it a slave. This will lead to missed writes and inconsistent state which depending on you application could be fine or maybe not.

To deal with this I have started to develop a "smarter" daemon to keep HAProxy in sync with the current master. My solution is at https://github.com/mdevilliers/redishappy.