Kubernetes Deployment makes great use of cpu and memory without stressing it

Question

I have deployed an app on Kubernetes and would like to test hpa. With kubectl top nodes command, i noticed that cpu and memory are increased without stressing it. Does it make sense?

Also while stressing deployment with apache bench, cpu and memory dont be increased enough to pass the target and make a replica.

My Deployment yaml file is so big to provide it. This is one of my containers.

    - name: web
      image: php_apache:1.0
      imagePullPolicy: Always
      resources:
        requests:
          memory: 50Mi
          cpu: 80m
        limits:
          memory: 100Mi
          cpu: 120m
      volumeMounts:
      - name: shared-data
        mountPath: /var/www/html
      ports:
      - containerPort: 80

It consists of 15 containers I have a VM that contains a cluster with 2 nodes (master,worker).

I would like to stress deployment so that i can see it scale up. But here I think there is a problem! Without stressing the app, the

CPU/Memory from Pod has passed the target and 2 replicas have been made (without stressing it). I know that the more Requests i provide to containers the less is that percentage. But does it make sense the usage of memory/cpu to be increased from the beggining, without stressing it?

I would like, the left part of target(the usage of memory in pods), be at the beggining 0% and as much as I stress it to be increased and create replicas. But as i'm stressing with apache bench, the value is increased by a maximum of 10%

We can see here the usage of CPU: kubectl top pods

NAME CPU(cores) MEMORY(bytes) x-app-55b54b6fc8-7dqjf 76m 765Mi

!!59% is the usage of memory from the pod and is described by Sum of Memory Requests/Memory(usage of memory). In my case 59% = 765Mi/1310Mi

HPA yaml file:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: hpa
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 35

Everything depends what you included in your configuration YAMLs. Could you provide those YAMLs? What application did you deployed, what it should do. Did you set CPU/Memory Request/Limits? What is your environment? How did you try to increase this load? You must provide more information as this question is too general. — PjoterS

PjoterS PjoterS · Accepted Answer · 2020-09-21T13:14:13

With kubectl top nodes command, i noticed that cpu and memory are increased without stressing it. Does it make sense?

Yes, it makes sense. If you will check Google Cloud about Requests and Limits

Requests and limits are the mechanisms Kubernetes uses to control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource. Limits, on the other hand, make sure a container never goes above a certain value. The container is only allowed to go up to the limit, and then it is restricted.

But does it make sense the usage of memory/cpu to be increased from the beggining, without stressing it?

Yes as, for example your container www it can start with memory: 50Mi and cpu: 80m but its allowed to increase to memory: 100Mi and cpu: 120m. Also as you mentioned you have 15 containers in total, so depends on their request, limits it can reach more than 35% of your memory.

In HPA documentation - algorithm-details you can find information:

When a targetAverageValue or targetAverageUtilization is specified, the currentMetricValue is computed by taking the average of the given metric across all Pods in the HorizontalPodAutoscaler's scale target. Before checking the tolerance and deciding on the final values, we take pod readiness and missing metrics into consideration, however.

All Pods with a deletion timestamp set (i.e. Pods in the process of being shut down) and all failed Pods are discarded.

If a particular Pod is missing metrics, it is set aside for later; Pods with missing metrics will be used to adjust the final scaling amount.

Not sure about last question:

!!59% is the usage of memory from the pod and is described by Sum of Memory Requests/Memory(usage of memory). In my case 59% = 765Mi/1310Mi

In your HPA you set to create another pod when averageUtilization: will reach 35% of memory. It reached 59% and it created another pod. As HPA target is memory, HPA is not counting CPU at all. Also please keep in mind as this is average it needs about ~1 minute to change values.

For better understanding how HPA is working, please try this walkthrough.

If this was not helpful, please clarify what are you exact asking.

Kubernetes Deployment makes great use of cpu and memory without stressing it

1 Answers