0
votes

Update 2: I was able to get the statistics by using grafana and influxDB. However, I find this overkill. I want to see the current status of my cluster, not persee the historical trends. Based on the linked image, it should be possible by using the pre-deployed Heapster and the Kubernetes Dashboard

Update 1: With the command below, I do see resource information. I guess the remaining part of the question is why it is not showing up (or how I should configure it to show up) in the kubernetes dashboard, as shown in this image: https://docs.giantswarm.io/img/dashboard-ui.png

$ kubectl top node
NAME                         CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
k8s-agentpool0-41204139-0    36m          1%        682Mi           9%        
k8s-agentpool0-41204139-1    33m          1%        732Mi           10%       
k8s-agentpool0-41204139-10   36m          1%        690Mi           10%  
[truncated]

I am trying to monitor performance in my Azure Kubernetes deployment. I noticed it has Heapster running by default. I did not launch this one, but do want to leverage it if it is there. My question is: how can I access it, or is there something wrong with it? Here are the details I can think of, let me know if you need more.

$ kubectl cluster-info
Kubernetes master is running at https://[hidden].uksouth.cloudapp.azure.com
Heapster is running at https://[hidden].uksouth.cloudapp.azure.com/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://[hidden].uksouth.cloudapp.azure.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubernetes-dashboard is running at https://[hidden].uksouth.cloudapp.azure.com/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
tiller-deploy is running at https://[hidden].uksouth.cloudapp.azure.com/api/v1/namespaces/kube-system/services/tiller-deploy:tiller/proxy

I set up a proxy:

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

Point my browser to

localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/workload?namespace=default

I see the kubernetes dashboard, but do notice that I do not see the performance graphs that are displayed at https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/. I also do not see the admin section.

I then point my browser to localhost:8001/api/v1/namespaces/kube-system/services/heapster/proxy and get

404 page not found

Inspecting the pods:

kubectl get pods --namespace=kube-system
NAME                                            READY     STATUS    RESTARTS   AGE
heapster-2205950583-43w4b                       2/2       Running   0          1d
kube-addon-manager-k8s-master-41204139-0        1/1       Running   0          1d
kube-apiserver-k8s-master-41204139-0            1/1       Running   0          1d
kube-controller-manager-k8s-master-41204139-0   1/1       Running   0          1d
kube-dns-v20-2000462293-1j20h                   3/3       Running   0          16h
kube-dns-v20-2000462293-hqwfn                   3/3       Running   0          16h
kube-proxy-0kwkf                                1/1       Running   0          1d
kube-proxy-13bh5                                1/1       Running   0          1d
[truncated]
kube-proxy-zfbb1                                1/1       Running   0          1d
kube-scheduler-k8s-master-41204139-0            1/1       Running   0          1d
kubernetes-dashboard-732940207-w7pt2            1/1       Running   0          1d
tiller-deploy-3007245560-4tk78                  1/1       Running   0          1d

Checking the log:

$kubectl logs heapster-2205950583-43w4b heapster  --namespace=kube-system
I0309 06:11:21.241752      19 heapster.go:72] /heapster --source=kubernetes.summary_api:""
I0309 06:11:21.241813      19 heapster.go:73] Heapster version v1.4.2
I0309 06:11:21.242310      19 configs.go:61] Using Kubernetes client with master "https://10.0.0.1:443" and version v1
I0309 06:11:21.242331      19 configs.go:62] Using kubelet port 10255
I0309 06:11:21.243557      19 heapster.go:196] Starting with Metric Sink
I0309 06:11:21.344547      19 heapster.go:106] Starting heapster on port 8082
E0309 14:14:05.000293      19 summary.go:389] Node k8s-agentpool0-41204139-32 is not ready
E0309 14:14:05.000331      19 summary.go:389] Node k8s-agentpool0-41204139-56 is not ready
[truncated the other agent pool messages saying not ready]
E0309 14:24:05.000645      19 summary.go:389] Node k8s-master-41204139-0 is not ready


$kubectl describe pod heapster-2205950583-43w4b  --namespace=kube-system
Name:           heapster-2205950583-43w4b
Namespace:      kube-system
Node:           k8s-agentpool0-41204139-54/10.240.0.11
Start Time:     Fri, 09 Mar 2018 07:11:15 +0100
Labels:         k8s-app=heapster
                pod-template-hash=2205950583
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"heapster-2205950583","uid":"ac75e772-2360-11e8-9e1c-00224807...
            scheduler.alpha.kubernetes.io/critical-pod=
Status:         Running
IP:             10.244.58.2
Controlled By:  ReplicaSet/heapster-2205950583
Containers:
  heapster:
    Container ID:  docker://a9205e7ab9070a1d1bdee4a1b93eb47339972ad979c4d35e7d6b59ac15a91817
    Image:         k8s-gcrio.azureedge.net/heapster-amd64:v1.4.2
    Image ID:      docker-pullable://k8s-gcrio.azureedge.net/heapster-amd64@sha256:f58ded16b56884eeb73b1ba256bcc489714570bacdeca43d4ba3b91ef9897b20
    Port:          <none>
    Command:
      /heapster
      --source=kubernetes.summary_api:""
    State:          Running
      Started:      Fri, 09 Mar 2018 07:11:20 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     121m
      memory:  464Mi
    Requests:
      cpu:        121m
      memory:     464Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from heapster-token-txk8b (ro)
  heapster-nanny:
    Container ID:  docker://68e021532a482f32abec844d6f9ea00a4a8232b8d1004b7df4199d2c7d3a3b4c
    Image:         k8s-gcrio.azureedge.net/addon-resizer:1.7
    Image ID:      docker-pullable://k8s-gcrio.azureedge.net/addon-resizer@sha256:dcec9a5c2e20b8df19f3e9eeb87d9054a9e94e71479b935d5cfdbede9ce15895
    Port:          <none>
    Command:
      /pod_nanny
      --cpu=80m
      --extra-cpu=0.5m
      --memory=140Mi
      --extra-memory=4Mi
      --threshold=5
      --deployment=heapster
      --container=heapster
      --poll-period=300000
      --estimator=exponential
    State:          Running
      Started:      Fri, 09 Mar 2018 07:11:18 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     50m
      memory:  90Mi
    Requests:
      cpu:     50m
      memory:  90Mi
    Environment:
      MY_POD_NAME:       heapster-2205950583-43w4b (v1:metadata.name)
      MY_POD_NAMESPACE:  kube-system (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from heapster-token-txk8b (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  heapster-token-txk8b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  heapster-token-txk8b
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     <none>
Events:          <none>
1

1 Answers

2
votes

I have seen in the past that if you restart the dashboard pod it starts working. Can you try that real fast and let me know?