Why make Min and Target Replica Size the same in Service Fabric?

Question

This question is specifically for Service Fabric but the concept goes beyond this one Cluster system so feel free to pitch in without SF experience.

I'm trying to understand the Pros and Cons of making MinReplicaSetSize and TargetReplicaSetSize the same or different numbers for a Stateful service.

Let's say you have 10 Partitions and 3 Replicas. This means every Partition would have 3 Replicas (1 Primary and 2 Secondary). Lets say you have some Collection distributed across these 10 Partitions. There are 2 situations to consider:

MinReplica and TargetReplica = 3

On some Partition the Primary Replica fails. One of the Secondaries is promoted to Primary. Since Replica count = 2 < MinReplica then writes to this Partition fail/halt until the new Secondary is spooled up and brought up to speed with the others so that the count is 3 again. The obvious downside is the halting of writes.

MinReplica = 2 and TargetReplica = 3

On some Partition the Primary Replica fails. One of the Secondaries is promoted to Primary. Since the MinReplica count is still safe, the writing to the new primary continues and it updates the 1 left Secondary. While this is happening another Secondary is spooled up and then must be brought up to speed while writes continue. So what is the downside?

Is it possible for the new Primary to fail, the Secondary to promote to Primary and also fail, before the new Secondary is caught up to speed; therefore losing committed data? Is this the downside? And in a MinReplica=3 / TargetReplica=4 scenario even this wouldn't happen.

I ask because I commonly see these as equal numbers and got thinking about it.

Answer 1

This statement "On some Partition the Primary Replica fails. One of the Secondaries is promoted to Primary. Since Replica count = 2 < MinReplica then writes to this Partition fail/halt until the new Secondary is spooled up and brought up to speed with the others so that the count is 3 again." is incorrect.

In order for writes to succeed, the operation must succeed on a majority (N/2)+1 of N replicas. Having three replicas, means quorum size is 2. If 2 writes succeed, the write operation succeeds. Changing it into 2 doesn't affect quorum. The size of the quorum is derived from the MinReplicaSetSize size. Changing it into 5 would mean a quorum size of 3, and then your statement about losing 1 replica would become valid.

Also, notice that the MinReplicaSetSize is a 'view' on the replica set:

The MinReplicaSetSize defines the minimum number of replicas in the view, so for example if the TargetReplicaSetSize is five and the MinReplicaSetSize is three, then even if there are three concurrent failures [it] will still have three replicas in its view of the replica set.