Kubernetes pods and shared memory and state

4
votes

I am a Kubernetes noobie so my apologies if this is a basic question - I am trying to get a good understanding of this amazing technology.

Let's say I have some processing on a pod and I have some variables and arrays. Now I will have a heavy load, and a new pod will be dispensed/replicated - now I have 2 pods rather than 1 to handle my load. What will happen to the memory of the second pod? Is pod 2 replicated with the same memory contents? will memory be shared? If there is some communication across my microservices which requires memory to be consistent, in other words there is some variables dependency, then replicating a pod with fresh memory is catastrophic. For clarifcation, let us say there is a key, value map on one pod. A pod is replicated, then we have two key,value map (one on pod1 and another on pod2). But we want a key,value map containing all the data that is common between these two pods - a shared key,value map across replicated pods - NOT two isolated key,value maps.

Sure, I suppose one solution would be to have a shared, central, database for reference. But this is a massive overhead for situations where we just want to share an array of a few hundred elements! What is the Kubernetes approach to this issue?

3
kubernetesgoogle-kubernetes-engine
This is an application-level design issue, not a Kubernetes thing. What would you do if you had 2+ replicas of your service that you had deployed manually? – Oliver Charlesworth
I would probably try to put some technique for communication across both replicas. But I don't know what that would manifest itself in. If this is not something to be handled at the Kubernetes level, then I would appreciate any comments on what would be good practice anyway. – Zeruno

3 Answers

1
votes

@Zeruno maybe i am thinking out loud here but what if you used the existing etcd cluster as a key-value store for your application

  1. No overhead of maintaining it since its already there and manged by K8s itself that too in a distributed cluster mode.

  2. This might act as a single point of storage that will be accessible by all the pods in that cluster.

1
votes

Addressing your question about memory of the second pod, the second pod will be assigned a new memory.

Addressing your question about the replicated memory content of the new pod, all the new pod will be replicated depending on the container spec.

Addressing your question about shared memory between pods, no the memory is not shared between the pods.If you want to share elements between pods, you need to make sure it is in container spec, such as volume or environment variables.

Addressing your question about key values, if you are looking for storage solutions, you could consider volumes 1, 2. If you are just looking to pass some key-values to pods, you might consider using configmaps.

1 https://kubernetes.io/docs/concepts/storage/persistent-volumes/

2 https://kubernetes.io/docs/concepts/storage/

0
votes

The solution you propose is appropriate sometimes, but I would only use it >1GB of shared data. In your case "array of a few hundred elements" I'd use file watcher on a PVC (volume), but my cluster has fast SSD arrays for shared storage, I couldn't recommend it if you're running on traditional disks or traditional SAN.

That said, >1GB shared memory isn't implemented yet in Kubernetes, as of 2020-04-28. See https://github.com/kubernetes/kubernetes/issues/28272 .

My reading of this thread is Kube limits shared memory to 64MB today. If your dataset is that small you could read changes off of shared storage quickly enough (I'd guess 200ms-3sec). If you need quicker I'd suggest using a caching service today: Redis with persistence disabled, or memcached, etc. and reconsider shared memory when issue 28272 is implemented.