Monitor only one namespace pods - Prometheus & Kubernetes & cadvisor

Question

I have deployed istio on kubernetes, and I installed prometheus from istio addons. My goal is to only monitor some pods of one application(such as all pods of bookinfo application). The job definition for monitoring pods is as below:

    - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      job_name: kubernetes-nodes-cadvisor
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        source_labels:
        - __meta_kubernetes_node_name
        target_label: __metrics_path__
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true

My problem is that I don't know how to monitor only one namespace's pods. For example, I deploy the bookinfo application in a namespace named Book. I only want the metrics of pods from namespace Book. However, prometheus will collect all pods metrics of the nodes. Instead of changing annotations of the application like Monitor only one namespace metrics - Prometheus with Kubernetes, I want know if there is a method to select only one namespace by changing the job definition above. Or is there some way to choose the monitor pods by it's labels?

Answer 1

I found a method in cadvisor's document. https://github.com/google/cadvisor/blob/master/docs/runtime_options.md It says that we can change the parameter '--docker_only' and '--raw_cgroup_prefix_whitelist' to choose the container to be monitored.

Answer 2

The following will match all target pods with label: some_label with any value.

relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_label_some_label]
    regex: (.*)

If you want to keep targets with label: monitor and value: true you would do:

relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_label_monitor]
    regex: true

All pods that don't match it will be dropped from scraping.

The same you should be able to do for namespaces:

relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_namespace]
    regex: Book

---

EDIT >

is there a way to change the [container_label_io_kubernetes_container_name] labels into "container_name"?

Try this:

relabel_configs:
  - action: replace
    source_labels: [container_label_io_kubernetes_container_name]
    target_label: container_name

It's all explained in prometheus docs about configuration