I am using Kafka Helm charts from here. I was trying Horizontal Pod Autoscaler for the same.
I added a hpa.yaml file as given below inside the templates folder.
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: kafka-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: {{ include "kafka.fullname" . }}
minReplicas: {{ .Values.replicas }}
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 50
- type: Resource
resource:
name: memory
targetAverageValue: 8000Mi
I have also tried the above YAML with kind: StatefulSet but the same issue persists.
My intention is to have 3 Kafka pods initially and scale it up to 5 based on CPU and memory targetValues as mentioned above.
However, the hpa gets deployed but it is unable to read the metrics as per my understanding as the current usage shows unknown as mentioned below.
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
kafka-hpa Deployment/whopping-walrus-kafka <unknown>/8000Mi, <unknown>/50% 3 5 0 1h .
I am new to helm and Kubernetes, so I am assuming there might be some issue with my understanding.
I have also deployed metrics-server.
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
metrics-server 1 1 1 1 1d
whopping-walrus-kafka-exporter 1 1 1 1 1h
Pods output
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
metrics-server-55cbf87bbb-vm2v5 1/1 Running 0 15m
whopping-walrus-kafka-0 1/1 Running 1 1h
whopping-walrus-kafka-1 1/1 Running 0 1h
whopping-walrus-kafka-2 1/1 Running 0 1h
whopping-walrus-kafka-exporter-5c66b5b4f9-mv5kv 1/1 Running 1 1h
whopping-walrus-zookeeper-0 1/1 Running 0 1h
I want the whopping-walrus-kafka pod to scale up to 5 on load, however, there's no deployment corresponding to it.
StatefulSet Output
$ kubectl get statefulset
NAME DESIRED CURRENT AGE
original-bobcat-kafka 3 2 2m
original-bobcat-zookeeper 1 1 2m
Output of describe hpa when kind in hpa.yaml is StatefulSet.
$ kubectl describe hpa
Name: kafka-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Fri, 18 Jan 2019 12:13:59 +0530
Reference: StatefulSet/original-bobcat-kafka
Metrics: ( current / target )
resource memory on pods: <unknown> / 8000Mi
resource cpu on pods (as a percentage of request): <unknown> / 5%
Min replicas: 3
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False FailedGetScale the HPA controller was unable to get the target's current scale: no matches for kind "StatefulSet" in group "extensions"
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 15s (x17 over 8m) horizontal-pod-autoscaler no matches for kind "StatefulSet" in group "extensions"
Output of describe hpa when kind in hpa.yaml is Deployment.
$ kubectl describe hpa
Name: kafka-hpa
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Fri, 18 Jan 2019 12:30:07 +0530
Reference: Deployment/good-elephant-kafka
Metrics: ( current / target )
resource memory on pods: <unknown> / 8000Mi
resource cpu on pods (as a percentage of request): <unknown> / 5%
Min replicas: 3
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False FailedGetScale the HPA controller was unable to get the target's current scale: could not fetch the scale for deployments.extensions good-elephant-kafka: deployments/scale.extensions "good-elephant-kafka" not found
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetScale 9s horizontal-pod-autoscaler could not fetch the scale for deployments.extensions good-elephant-kafka: deployments/scale.extensions "good-elephant-kafka" not found
Output from metrics server pod
$ kubectl describe pods metrics-server-55cbf87bbb-vm2v5
Name: metrics-server-55cbf87bbb-vm2v5
Namespace: default
Node: docker-for-desktop/192.168.65.3
Start Time: Fri, 18 Jan 2019 11:26:33 +0530
Labels: app=metrics-server
pod-template-hash=1176943666
release=metrics-server
Annotations: <none>
Status: Running
IP: 10.1.0.119
Controlled By: ReplicaSet/metrics-server-55cbf87bbb
Containers:
metrics-server:
Container ID: docker://ee4b3d9ed1b15c2c8783345b0ffbbc565ad25f1493dec0148f245c9581443631
Image: gcr.io/google_containers/metrics-server-amd64:v0.3.1
Image ID: docker-pullable://gcr.io/google_containers/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b
Port: <none>
Host Port: <none>
Command:
/metrics-server
--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP
--logtostderr
State: Running
Started: Fri, 18 Jan 2019 11:26:35 +0530
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-d2g7b (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
metrics-server-token-d2g7b:
Type: Secret (a volume populated by a Secret)
SecretName: metrics-server-token-d2g7b
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
Guys please feel free to clear my understanding as well if I am going wrong somewhere.
Help on this would be highly appreciated.
resources: requests: memory: "64Mi" cpu: "250m"are set for your statefulset. You can check that usingkubectl describe statefulset <sf-name>If not then that can be one reason. – Prafull Ladha