Redis replication for large data to new slave

0
votes

I have a redis master which has 30 GB of data and the memory there is 90 GB. We have this setup as we have less writes and more reads. Normally, we would have a 3X db size RAM machine.

The problem here is, one slave went corrupt and later on when we added it back using sentinel. it got stuck in wait_bgsave state on master (after seeing the info on master)

The reason was that : 

 client-output-buffer-limit slave 256mb 64mb 60

This was set on master and since max memory is not available it breaks replication for the new slave. I saw this question Redis replication and client-output-buffer-limit where similar issue is being discussed but i have a broader scope of question.

We can't use a lot of memory. So, what are the possible ways to do replication in this context to prevent any failure on master (wrt. memory and latency impacts)

I have few things on mind: 1 - Should i do diskless replication - will it have any impact on latency of writes and reads? 2 - Should i just copy the dump file from another slave to this new slave and restart redis. ? will that work. 3 - Should i increase the output-buffer-limit slave to a greater limit? If yes, then how much? I want to do this for sometime till replication happens and then revert it back to normal setting? I am skeptic about this approach.

1
redisreplication

1 Answers

1
votes

You got this problem, because you have a slow replica, and it cannot read the replication data as fast as needed.

In order to solve the problem, you can try to increase the client-output-buffer-limit buffer limit. Also you can try to disable persistence on replica when it syncing from master, and enable persistence after that. By disabling persistence, replica might consume the data faster. However, if the bandwidth between master and replica is really small, you might need to consider re-deploy your replica to make it near the master, and have a large bandwidth.

1 - Should i do diskless replication - will it have any impact on latency of writes and reads?

IMHO, I think it has nothing to do with diskless replication.

2 - Should i just copy the dump file from another slave to this new slave and restart redis. ? will that work.

NO, it won't work.

3 - Should i increase the output-buffer-limit slave to a greater limit? If yes, then how much? I want to do this for sometime till replication happens and then revert it back to normal setting?

YES, you can try to increase the limit. And in your case, since your data size is 30G, so a hard limit of 30G should slove the problem. However, that's too much, and might have other impact. You need to do some benchmark to get a right limit.

YES, you can dynamically change this setting by the CONFIG SET command.