How to add resource and limits on Kubernetes Engine on Google Cloud Platform

Question

I am trying to add resource and limits to my deployment on Kuberenetes Engine since one of my deployment on the pod is continuously getting evicted with an error message The node was low on resource: memory. Container model-run was using 1904944Ki, which exceeds its request of 0. I assume that the issue could be resolved by adding resource requests.

When I try to add resource requests and deploy, the deployment is successful but when I go back and and view detailed information about the Pod, with the command kubectl get pod default-pod-name --output=yaml --namespace=default It still says the pod has request of cpu: 100m and without any mention of memory that I have allotted. I am guessing that the cpu request of 100m was a default one. Please let me know how I can allot the requests and limits, the code I am using to deploy is as follows:

kubectl run model-run --image-pull-policy=Always --overrides='
{
    "apiVersion": "apps/v1beta1",
    "kind": "Deployment",
    "metadata": {
        "name": "model-run",
        "labels": {
            "app": "model-run"
        }
    },
    "spec": {
        "selector": {
            "matchLabels": {
                "app": "model-run"
            }
        },
        "template": {
            "metadata": {
                "labels": {
                    "app": "model-run"
                }
            },
            "spec": {
                "containers": [
                    {
                        "name": "model-run",
                        "image": "gcr.io/some-project/news/model-run:development",
                    "imagePullPolicy": "Always",
                      "resouces": {
                        "requests": [
                          {
                            "memory": "2048Mi",
                            "cpu": "500m"
                          }
                        ],
                        "limits": [
                          {
                            "memory": "2500Mi",
                            "cpu": "750m"
                          }
                        ]
                      },
                        "volumeMounts": [
                            {
                                "name": "credentials",
                                "readOnly": true,
                                "mountPath":"/path/collection/keys"
                            }
                        ],
                        "env":[
                            {
                                "name":"GOOGLE_APPLICATION_CREDENTIALS",
                                "value":"/path/collection/keys/key.json"
                            }
                                ]
                    }
                ],
                "volumes": [
                    {
                        "name": "credentials",
                        "secret": {
                            "secretName": "credentials"
                        }
                    }
                ]
            }
        }
    }
}
'  --image=gcr.io/some-project/news/model-run:development

Any solution will be appreciated

@koe sorry the typo was while posting it here. The issue still persists — Ranga Vittal

Will R.O.F. Will R.O.F. · Accepted Answer · 2020-05-19T13:33:09

The node was low on resource: memory. Container model-run was using 1904944Ki, which exceeds its request of 0.

At first the message seems like there is a lack of resource in the node itself but the second part makes me believe you are correct in trying to raise the request limit for the container.

Just keep in mind that if you still face errors after this change, you might need to add mode powerful node-pools to your cluster.

I went through your command, there is a few issues I'd like to highlight:

kubectl run was deprecated in 1.12 to all resources except for pods and it is retired in version 1.18.
apiVersion": "apps/v1beta1 is deprecated, and starting on v 1.16 it is no longer be supported, I replaced with apps/v1.
In spec.template.spec.container it's written "resouces" instead of "resources"
after fixing the resources the next issue is that requests and limits are written in array format, but they need to be in a list, otherwise you get this error:

kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
error: v1beta1.Deployment.Spec: v1beta1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Resources: v1.ResourceRequirements.Limits: ReadMapCB: expect { or n, but found [, error found in #10 byte of ...|"limits":[{"cpu":"75|..., bigger context ...|Always","name":"model-run","resources":{"limits":[{"cpu":"750m","memory":"2500Mi"}],"requests":[{"cp|...

Here is the fixed format of your command:

kubectl run model-run --image-pull-policy=Always --overrides='{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "metadata": {
    "name": "model-run",
    "labels": {
      "app": "model-run"
    }
  },
  "spec": {
    "selector": {
      "matchLabels": {
        "app": "model-run"
      }
    },
    "template": {
      "metadata": {
        "labels": {
          "app": "model-run"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "model-run",
            "image": "nginx",
            "imagePullPolicy": "Always",
            "resources": {
              "requests": {
                "memory": "2048Mi",
                "cpu": "500m"
              },
              "limits": {
                "memory": "2500Mi",
                "cpu": "750m"
              }
            },
            "volumeMounts": [
              {
                "name": "credentials",
                "readOnly": true,
                "mountPath": "/path/collection/keys"
              }
            ],
            "env": [
              {
                "name": "GOOGLE_APPLICATION_CREDENTIALS",
                "value": "/path/collection/keys/key.json"
              }
            ]
          }
        ],
        "volumes": [
          {
            "name": "credentials",
            "secret": {
              "secretName": "credentials"
            }
          }
        ]
      }
    }
  }
}'  --image=gcr.io/some-project/news/model-run:development

Now after aplying it on my Kubernetes Engine Cluster v1.15.11-gke.13 , here is the output of kubectl get pod X -o yaml:

$ kubectl get pods
NAME                         READY   STATUS    RESTARTS   AGE
model-run-7bd8d79c7d-brmrw   1/1     Running   0          17s

$ kubectl get pod model-run-7bd8d79c7d-brmrw -o yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: model-run
    pod-template-hash: 7bd8d79c7d
    run: model-run
  name: model-run-7bd8d79c7d-brmrw
  namespace: default
spec:
  containers:
  - env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /path/collection/keys/key.json
    image: nginx
    imagePullPolicy: Always
    name: model-run
    resources:
      limits:
        cpu: 750m
        memory: 2500Mi
      requests:
        cpu: 500m
        memory: 2Gi
    volumeMounts:
    - mountPath: /path/collection/keys
      name: credentials
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-tjn5t
      readOnly: true
  nodeName: gke-cluster-115-default-pool-abca4833-4jtx
  restartPolicy: Always
  volumes:
  - name: credentials
    secret:
      defaultMode: 420
      secretName: credentials

You can see that the resources limits and requests were set.

If you still have any question let me know in the comments!

How to add resource and limits on Kubernetes Engine on Google Cloud Platform

2 Answers