Service fabric reliable collections data loss after service upgrade

Question

Why reliable collections is empty after micro-service upgrade and not invoking event OnDataLossAsync to restore state from external backup?

We have large scale system based on stateful services

<StatefulServiceType ServiceTypeName="UserServiceType" HasPersistedState="true" />

HasPersistedState is set as true, and data replicated across replicas, in case of VM failure data still valid and recovering with OnDataLossAsync but after upgrade collections is empty.

I have tried all upgrade options (remove, keep, auto ugrade) application, result the same - collections is empty.

For now we decided to replicate data to blob storage and recover it after service update which is not perfect solution, data recovery takes a few minutes and it makes some service unavailable/inconsistent for that time.

So we are looking for solution that allows to save data after upgrade.

Answer 1

Upgrading a stateful service does not remove data in a Reliable Collection by design. The entire Service Fabric platform is built around solving this very problem with rolling in-place upgrades of stateful service.

My guess is you are either not actually performing rolling upgrades, and instead you are deleting and re-creating the service. That's what the default "Remove Application" setting in Visual Studio Application Debug Mode does - it deletes the application when you stop debugging (thus deleting state). The "Auto Upgrade" option will not delete the application and will automatically perform a rolling upgrading for you the next time you press F5. If your state is gone in this case, then I'm willing to bet there is a bug in your service code. Try it with a brand new stateful service project.