How to deploy pods across all nodes evenly in Kubernetes?

2
votes

I have a few Kubernetes clusters with different #of nodes in each. And my deployment of config has "replicas: #nodes". There is no specific config set up for scheduling that pod but after deployment, I see strange behavior in terms of the distribution of pods on nodes.

Example:

For 30 nodes cluster (30 replicas) all 30 pod replicas distributed across 25 nodes only and other 5 nodes are sitting ideal in the cluster. Similar cases for many other different clusters and this count varies in every new/redeployment.

Question:

I want to distribute my pod replicas across all nodes. If I set "replicas: #nodes" then I should have one pod replica in each node. If I increase/double the replicas count then it should distribute evenly. is there any specific configuration in deployment yaml for Kubernetes?

Configuration with node AntiAffinity, but still this one is behaving as above. Tried with "requiredDuringSchedulingIgnoredDuringExecution" and that one did deployed one pod in each node, but if I increase the replicas or any node goes down during the deployment then then whole deployment fails.

metadata:
  labels:
    app: test1
spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - test1
          topologyKey: kubernetes.io/hostname
2
kuberneteskube-scheduler
What problems do you experience with that pods is not evenly distributed? For what reason is it important to you that pods is scheduled to all nodes? Kubernetes abstract away the infrastructure so you should not think too much of this, unless you have a problem with it. – Jonas
What you're looking for is probably something like podAntiAffinity with preferredDuringSchedulingIgnoredDuringExecution which weighs the scheduling towards not putting a pod on a node where pods with a certain label already exists. It does not fail scheduling if it's not possible to schedule on a different node, just prefers a bit extra not to. – Joachim Isaksson
@Jonas- I faced the worst distribution of pods, 32 pods distributed across only 3 nodes instead of having 32 nodes cluster. – CRP08
@JoachimIsaksson- Already tried with above configuration you suggested but it, still behaving as I gave example in question. In my config where I want to prefer node not having similar pod label. But still it's distributed across 28 nodes instead of 32. I don't have any other pod running on those nodes which can cause the problem of resource usages. – CRP08
Can you paste the logs from kube-scheduler? See e.g. stackoverflow.com/questions/58885793/… – Jonas

2 Answers

3
votes

If you need a 1 pod per node deployment, you should be using a daemonSet instead of a replicaSet. Alternately, if you need more than 1 pod per node and still want the pod distribution to be mostly even, you can use pod anti-affinity as I discussed in this post

0
votes

see Pod Topology Spread Constraints https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ this feature gives you permission to finely define how your pods will be distributed across your cluster based on regions, zones, nodes, and other user-defined topology domains.

So you may define your own rules how to distribute pods.