Redis master slave sync infinite loop

Question

Our slave master replication keeps killing our redis connection. It tries to connect to the master and replicate the master data. Unfortunately the link to the master seems to fail and the full resync starts again instead of just a partial resync.

It's in a full resync loop which is killing the network andmaking redis unavailable

When I first searched for a a spolution I found some blog posts saying to increase the repl-timeout. This just increased the time between the servers going down. I increased it to 360.

Would a further, larger increase in time solve this issue or is the connection to the master being lost during the background save because of another setting?

Answer 1

Try to increase the "client-output-buffer-limit slave" hard and soft limits. During replication, when the hard limit is hit or a soft limit is hit and continues to stay above that limit for a specified number of seconds, the client is disconnected from the master and the sync is restarted.

client-output-buffer-limit Source: http://download.redis.io/redis-stable/redis.conf

Answer 2

Not a solution, but that was the problem in my case, might help to someone.

My slave machine had older redis installed but new RDB db, so each time it tried to resync with master, it failed with this error in logs: 11212:S 17 Oct 11:29:58.607 # Can't handle RDB format version 8 11212:S 17 Oct 11:29:58.607 # Failed trying to load the MASTER synchronization DB from disk

So I deleted the old RDB file from slave and that solved the problem. Always check the logs, guys. :)