How to get real-time resource usage of a pod in k8s?

1
votes

I'm setting up a service, which use k8s to start jupyter pod for each user. I need to provide a real-time resource usage of the jupyter pod when user query for it. I tried metrics-server and kubectl top. Their results are from cache and have about 1 minute delay. Is there any solution?

> kubectl version

Client Version: version.Info{Major:"1", Minor:"14",GitVersion:"v1.14.0",GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z",GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.6-aliyun.1", GitCommit:"8cb561c", GitTreeState:"", BuildDate:"2019-05-23T11:28:17Z", GoVersion:"go1.10.8", Compiler:"gc",Platform:"linux/amd64"}

1
kubernetes
You can talk directly to the cadvisor API, but there's some metrics which only make sense in batched form so they are usually aggregated over some short period of time. – coderanger
could you please add kubectl version? – Abu Hanifa
@coderanger no: cAdvisor has been deprecated since 1.12 even still available, any solution implementing this should not be recommended – prometherion

1 Answers

0
votes

you could query the kubelet stats endpoint:

curl --insecure https://<node url>:10250/stats/summary

you can also make the query more specific to pod/container

curl --insecure https://<node url>:10250/{namespace}/{podName}/{uid}/{containerName}

where uid is basically any string...

The part of the code for your version is here

Another tip: if your pod run as part of host network you could query localhost also and would need to assign a service account with access to it. The query would look like this:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl https://localhost:10250/stats/summary --header "Authorization: Bearer $TOKEN" --insecure