StatefulServiceBase.RunAsync called more than once in Primary during startup?

Question

Is StatefulServiceBase.RunAsync(CancellationToken) method called more than once in Primary during startup?

The documentation ([1] & [2]) says that this method will be called only in Primary but nothing about how many times it will be called during startup.

References:
[1] https://docs.microsoft.com/en-us/dotnet/api/microsoft.servicefabric.services.runtime.statefulservicebase.runasync?view=azure-dotnet
[2] https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-lifecycle#stateful-service-startup

Answer 1

The docs suggests:

RunAsync method is implemented as a processing loop and will only be called when the replica is primary with write status.

But what it does not make clear is that RunAsync() is not called only at startup, also called on ChangeRole event when a Secondary replica is promoted to a Primary.

Technically, it is called once at startup when the replica becomes primary, and if for any reason it is demoted to secondary by ChangeRole event and then promoted again for primary, it will be called again, each time the cancellation token will be cancelled to propagate the changes before new calls.

So in summary, it is often called once but should not be expected to do so, as it might be called multiple times, and will be an issue if the logic in there expect to always run once.

Other common problem is the CreateServiceReplicaListeners() can also be called multiple times, for the same reasons above, but can be a much bigger issue, because secondary replicas also call CreateServiceReplicaListeners(), a common mistake is add logic in there assuming it is called only once.

The docs here and here show more details about the issues with scenario above.

This is the source file for most of the logic described above.