How to expose kubernetes metric server api to curl from inside the pod?

Question

I am using metric server to get the usage of my Kubernetes cluster. But in order to use it from outside the host, I need to use "kubectl proxy". But i don't want to do that as it is not intended to run on background. I want it to be run as a service continuously

How can i achieve these

expected output curl clusterip:8001/apis/metrics.k8s.io/v1beta1/nodes

{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": [
    {
      "metadata": {
        "name": "manhattan-master",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/manhattan-master",
        "creationTimestamp": "2019-11-15T04:26:47Z"
      },
      "timestamp": "2019-11-15T04:26:33Z",
      "window": "30s",
      "usage": {
        "cpu": "222998424n",
        "memory": "3580660Ki"
      }
    }
  ]

I tried by using LoadBalancig service metrics-server-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: main-port
  externalTrafficPolicy: Local
  type: LoadBalancer

kubectl describe service metrics-master -n kube-system

[root@manhattan-master 1.8+]# kubectl describe service metrics-server -n kube-system
Name:                     metrics-server
Namespace:                kube-system
Labels:                   kubernetes.io/cluster-service=true
                          kubernetes.io/name=Metrics-server
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true","kubernetes.io/name":"Me...
Selector:                 k8s-app=metrics-server
Type:                     LoadBalancer
IP:                       10.110.223.216
Port:                     <unset>  443/TCP
TargetPort:               main-port/TCP
NodePort:                 <unset>  31043/TCP
Endpoints:                10.32.0.7:4443
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     32208
Events:                   <none>

I used service name "loadbalancer" instead of "clusterIp" in service.yaml file. It will give you loadbalancer endpoint and then you can use it. — Shree Prakash
I tried but i didn't get any ExternalIp in that case and my top command also failing — UDIT JOSHI
Hi, you can use your apiserver fqdn, curl -kv $API_SERVER_FQDN/apis/metrics.k8s.io/v1beta1/nodes, that's how kubectl get the information about usage — Suresh Vishnoi
@UDITJOSHI it should be there , just do kubectl get svc -n kube-system or in which namespace you deployed metric server — Shree Prakash
@ShreePrakash I have added the description of my metric-server service. Can you please tell me what wrong am i doing? — UDIT JOSHI

Mark Watney Mark Watney · Accepted Answer · 2019-11-19T10:47:53

This is possible by creating a new service to expose the Metrics Server. Your Metrics Server Service should look like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    kubernetes.io/name: Metrics-server-ext
  name: metrics-server-ext
  namespace: kube-system
  selfLink: /api/v1/namespaces/kube-system/services/metrics-server
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
  sessionAffinity: None
  type: LoadBalancer

If you try to access this service you will face some problems with authorization and you need to do some things to give all necessary authorizations.

After creating the service you will need to create a Cluster Role Binding so our service can have access to the data:

$ kubectl create clusterrolebinding node-admin-default-svc --clusterrole=cluster-admin --serviceaccount=default:default

Before running curl command we need to get the token so we can pass this on our curl command:

$ TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode)

Get your service external IP:

kubectl get svc/metrics-server-ext -n kube-system -o jsonpath='{..ip}'

Your curl command should pass the Token key to get Authorization:

curl -k https://34.89.228.98/apis/metrics.k8s.io/v1beta1/nodes --header "Authorization: Bearer $TOKEN" --insecure

Sample output:

{
 "kind": "NodeMetricsList",
 "apiVersion": "metrics.k8s.io/v1beta1",
 "metadata": {
   "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
 },
 "items": [
   {
     "metadata": {
       "name": "gke-lab-default-pool-993de7d7-ntmc",
       "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-lab-default-pool-993de7d7-ntmc",
       "creationTimestamp": "2019-11-19T10:26:52Z"
     },
     "timestamp": "2019-11-19T10:26:17Z",
     "window": "30s",
     "usage": {
       "cpu": "52046272n",
       "memory": "686768Ki"
     }
   },
   {
     "metadata": {
       "name": "gke-lab-default-pool-993de7d7-tkj9",
       "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-lab-default-pool-993de7d7-tkj9",
       "creationTimestamp": "2019-11-19T10:26:52Z"
     },
     "timestamp": "2019-11-19T10:26:21Z",
     "window": "30s",
     "usage": {
       "cpu": "52320505n",
       "memory": "687252Ki"
     }
   },
   {
     "metadata": {
       "name": "gke-lab-default-pool-993de7d7-v7m3",
       "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/gke-lab-default-pool-993de7d7-v7m3",
       "creationTimestamp": "2019-11-19T10:26:52Z"
     },
     "timestamp": "2019-11-19T10:26:17Z",
     "window": "30s",
     "usage": {
       "cpu": "45602403n",
       "memory": "609968Ki"
     }
   }
 ]
}

EDIT:

You can also optionally access it from your pods since you created a Cluster Role Binding in your default Service Account with cluster-admin role.

As example, create a pod from a image that includes curl command:

$ kubectl run bb-$RANDOM --rm -i --image=ellerbrock/alpine-bash-curl-ssl --restart=Never --tty -- /bin/bash

Than you need to exec into you pod and run:

$ curl -k -X GET https://kubernetes.default/apis/metrics.k8s.io/v1beta1/nodes --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --insecure

Here we are passing the same TOKEN mentioned before in a complete different way.

How to expose kubernetes metric server api to curl from inside the pod?

1 Answers