Is keep logging messages in group communication service or paxos practical?

3
votes

In the case of network partition or node crash, most of the distributed atomic broadcast protocols (like Extended Virtual Synchrony or Paxos), require running nodes, to keep logging messages, until the crashed or partitioned node rejoins the cluster. When a node rejoins the cluster, replay of logged messages are enough to regain the current state.

My question is, if the partitioned/crash node takes really long time to join the cluster again, then eventually logs will overflow. This seem to be a very practical issue, but still no one in their paper talks about it. Is there a very obvious solution to this which I am missing? Or my understanding in incorrect.

2
distributed-computingdistributedpaxos
"... no one in their paper talks about it." Are you asking about a specific paper? Or that no one ever seems to address this issue? – ad absurdum
I have read paper mostly related to Group Communication Services and there I have not found this been raised. For example, most of the Ken Birman's Virtual Synhcrony Papers, Extended Virtual Syncrhony, CoREL etc. – Satish Kumar

2 Answers

2
votes

You don't really need to remember the whole log. Imagine for example that the state you were synchronizing between the nodes was something like an SQL table with a row of the form (id: int, name: string) and the commands that would be written into the logs were in a form "insert row with id=x and name=y", "delete row where id=z", "set name=a where id=1000",...

Once such commands were committed, all you really care about is the final table. Then once a node which was offline for a long time goes online, it would only need to download the table + few entries from the log that were committed while the table was being downloaded.

This is called "log compaction", check out the chapter 7 in the Raft paper for more info.

2
votes

There are a few potential solutions to the infinite log problem but one of the more popular ones for replicated state machines is to periodically snap-shot the full replicated state machine and delete all history prior to that point. A node that has been offline too long would then just discard all of their information, download the snapshot, and start replaying the replicated logs from that point.