Service Fabric stateful services in-memory vs external storage

Question

I have a Service Fabric application and it contains two services stateless and stateful. Service Fabric Application Stateless Service: It contains API endpoints to communicate with stateful service. Stateful Service: The data is being stored in Reliable collections i.e in-memory storage.

I have around 15 service fabric microservices that will be communicating with each other based on the requirement. I'm ending up with a lot of proxy calls in order to communicate between the services which is one of the major reasons for performance hindrance.

In order to mitigate this issue, I have a thought to remove stateful service( in-memory storage with Reliable Dictionaries) and use external storage like Azure Cosmos DB as a data storage.

In the new approach, my application will have one stateless service and it will communicate with the external data store ( ex: Cosmo DB). Service Fabric Application Stateless Service: It contains API endpoints to communicate with the storage provider ( Ex: CosmosDB).

Can anyone let us know whether Service fabric in-memory or external storage gives more performance?

Apart from the performance issues with the in-memory storage, it is becoming very challenging to implement the complex queries or do any elastic search or creating reports as we have dependencies between the services.

Is there any other better approach that can really resolve these kinds of issues?

Answer 1

The whole point of using stateful services is to bring the data to where the compute (your service) is. The benefit of this is performance, as there is no network latency for getting the data.

Now, what you are doing is effectively throwing this benefit away by using a stateful service as a central datastore for other services to get data from.

There are at least two option I can think of. The first is to use an external datastore like Cosmos DB and have all services connect to that datastore or, second opion, to convert your stateless services to stateful services and copy/distribute only the portions of the data a given service need to that service. To make it easier to report based on the data you could create read models.

Currently, we have a databse and moving all databse tables as microservices. Inorder to implement stored procedures/ views, we are fetching few services data in a single service and implemting the logic. Do we have an alternative approach for the Sp's/ Views?

You should not try to map a database and its views/stored procedure to some logic and microservices. Instead, try a new view on it. Let each service put their own data into one or more reliable collections. If there is need for a data store with data combined from each service have those services update a so called read model (you'll probably and up having more than one readmodel).

Look up terms like CQRS and read models, they will help with a micro services architecure.

Or have all services connect to, for example, a sql server giving the benefits of stored procedures and views. But do mind that once you use a centralized database, whether it is a sql database or cosmos db database, your micro services are no longer independent services as they all share a single database schema.