Sharded load balancing for stateful services in Kubernetes

Question

I am currently switching from Service Fabric to Kubernetes and was wondering how to do custom and more complex load balancing.

So far I already read about Kubernetes offering "Services" which do load balancing for pods hidden behind them, but this is only available in more plain ways.

What I want to rewrite right now looks like the following in Service Fabric:

I have this interface:

public interface IEndpointSelector
{
    int HashableIdentifier { get; }
}

A context keeping track of the account in my ASP.Net application e.g. inherits this. Then, I wrote some code which would as of now do service discovery through the service fabric cluster API and keep track of all services, updating them when any instances die or are being respawned.

Then, based on the deterministic nature of this identifier (due to the context being cached etc.) and given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.

Now, how would I go about doing this in Kubernetes?

As I already mentioned, I found "Services", but it seems like their load balancing does not support custom logic and is rather only useful when working with stateless instances.

Is there also a way to have service discovery in Kubernetes which I could use here to replace my existing code at some points?

Jonas Jonas · Accepted Answer · 2019-11-03T08:45:52

StatefulSet

StatefulSet is a building block for stateful workload on Kubernetes with certain guarantees.

Stable and unique network identity

StatefulSet Pods have a unique identity that is comprised of an ordinal, a stable network identity, and stable storage.

As an example, if your StatefulSet has the name sharded-svc

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sharded-svc

And you have e.g. 3 replicas, those will be named by <name>-<ordinal> where ordinal starts from 0 up to replicas-1.

The name of your pods will be:

sharded-svc-0
sharded-svc-1
sharded-svc-2

and those pods can be reached with a dns-name:

sharded-svc-0.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-1.sharded-svc.your-namespace.svc.cluster.local
sharded-svc-2.sharded-svc.your-namespace.svc.cluster.local

given that your Headless Service is named sharded-svc and you deploy it in namespace your-namespace.

Sharding or Partitioning

given multiple replicas of the target service of a frontend -> backend call, I can reliably route traffic for a certain account to a certain endpoint instance.

What you describe here is that your stateful service is what is called sharded or partitioned. This does not come out of the box from Kubernetes, but you have all the needed building blocks for this kind of service. It may happen that it exists an 3rd party service providing this feature that you can deploy, or it can be developed.

Sharding Proxy

You can create a service sharding-proxy consisting of one of more pods (possibly from Deployment since it can be stateless). This app need to watch the pods/service/endpoints in your sharded-svc to know where it can route traffic. This can be developed using client-go or other alternatives.

This service implements the logic you want in your sharding, e.g. account-nr modulus 3 is routed to the corresponding pod ordinal

Update: There are 3rd party proxies with sharding functionallity, e.g. Weaver Proxy

Sharding request based on headers/path/body fields

Recommended reading: Weaver: Sharding with simplicity

Consuming sharded service

To consume your sharded service, the clients send request to your sharding-proxy that then apply your routing or sharding logic (e.g. request with account-nr modulus 3 is routed to the corresponding pod ordinal) and forward the request to the replica of sharded-svc that match your logic.

Alternative Solutions

Directory Service: It is probably easier to implement sharded-proxy as a directory service but it depends on your requirements. The clients can ask your directory service to what statefulSet replica should I send account-nr X and your serice reply with e.g. sharded-svc-2

Routing logic in client: The probably most easy solution is to have your routing logic in the client, and let this logic calculate to what statefulSet replica to send the request.