What am I trying to achieve?
Trying to install a horizontal pod scaler for one of our service in eks which has ALB infront of it. I used ALB ingress controller and ingress to spin up a load balancer. Does it have any impact on geting metrics?
What is NOT working?
When I use cpu/memory Utilization values in the metric to use for scaling, the utilization metric cannot be find.
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
What is Working?
But if I use value/AverageValue of cpu (ex: 200m etc) it is able to get the metric and working as expected.
metrics:
- type: Resource
resource:
name: cpu
target:
type: AverageValue
averageValue: 500m
What I have tried?
Metrics server in kube-system.
NAME READY UP-TO-DATE AVAILABLE AGE metrics-server 1/1 1 1 11h
Hack that I used to avoid access issues
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
Service deployment with resource requests
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
With above components I am able to get metrics of nodes, pods etc with out any errors.
kubectl top pods
NAME CPU(cores) MEMORY(bytes)
xxxxxxxxxxxxxxx 5m 161Mi
xxxxxxxxxxxxxxx 3m 132Mi
xxxxxxxxxxxxxxx 4m 149Mi
I am not getting percentages here as well, assuming we wont get it here.
I enabled the log level v=4 to metrics server and I couldnt see any issues.
kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Error message I see is
Name: xxxxxxx
Namespace: xxxxxxx
Labels: app=xxxxxxx
Annotations: <none>
CreationTimestamp: Tue, 29 Sep 2020 21:09:58 -0400
Reference: Deployment/xxxxxxxx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 50%
Min replicas: 2
Max replicas: 8
Deployment pods: 2 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: missing request for cpu
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetResourceMetric 5m37s (x320 over 100m) horizontal-pod-autoscaler missing request for cpu
Warning FailedComputeMetricsReplicas 37s (x340 over 100m) horizontal-pod-autoscaler invalid metrics (1 invalid out of 1), first error is: failed to get cpu utilization: missing request for cpu
As I am able to scale it using Value/AverageValue but not Utilization confuses me, am I missing anything here ?
