Unable to retrieve custom metrics from prometheus-adapter

Question

i am trying to experiment with scaling one of my application pods running on my raspberry pi kubernetes cluster using HPA + custom metrics but ran into several issues which despite reading the documentations on https://github.com/DirectXMan12/k8s-prometheus-adapter and troubleshooting for the past 2 days, i am still having difficulties grasping why some problems are happening.

Firstly, i built an ARM-compatible image of k8s-prometheus-adapter and install it using helm. I can confirm its running properly by checking the pod logs.

I have also set up a script which sends raspberry pis temperature to pushgateway and i can query via this Prometheus query node_temp, which will return the following series

node_temp{job="kube4"}  42
node_temp{job="kube1"}  44
node_temp{job="kube2"}  39
node_temp{job="kube3"}  40

Now i want to be able to scale one of my application pods using the above temperature values as an experiment to understand better how it works.

Below is my k8s-prometheus-adapter helm values.yml file

image:
  repository: jaanhio/k8s-prometheus-adapter-arm
  tag: latest
logLevel: 7
prometheus:
  url: http://10.17.0.12
rules:
  default: false
  custom:
  - seriesQuery: 'etcd_object_counts'
    resources:
      template: <<.Resource>>
    name:
      as: "etcd_object"
    metricsQuery: count(etcd_object_counts)
  - seriesQuery: 'node_temp'
    resources:
      template: <<.Resource>>
    name:
      as: "node_temp"
    metricsQuery: count(node_temp)

After installing via helm, i ran kubectl get apiservices and can see v1beta1.custom.metrics.k8s.io listed.

i then ran kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq and got the following

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "jobs.batch/node_temp",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "jobs.batch/etcd_object",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
   ]

i then tried to query the value of the registered node_temp metrics using kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp but got the following response Error from server (InternalError): Internal error occurred: unable to list matching resources

Questions:

Why is the node_temp metrics associated with jobs.batch resource type?
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/foo_metrics

Any help and advice will be greatly appreciate!

shipperizer shipperizer · Accepted Answer · 2020-12-04T15:06:32

Why is the node_temp metrics associated with jobs.batch resource type?

It picks the labels attached to the prometheus metrics and tries to interpret them, in this case u have clearely "job-kube4"

Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?

Metrics are namespaced, see the "namespaced:true" so you'll need "/apis/custom.metrics.k8s.io/v1beta1/namespaces//jobs//node_temp"

What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs//node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces//metrics/foo_metrics

Check https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md#api-paths

Unable to retrieve custom metrics from prometheus-adapter

1 Answers