Duplicate scrape targets on different ports

Question

I'm using a scrape config that is very similar to the suggested ones in the examples. For some reason I have duplicate scrape targets. I'd like to know why this is the case

My scrape config:

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_scrape ]
        action: keep
        regex: true
      - source_labels: [ __meta_kubernetes_pod_annotation_prometheus_io_path ]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [ __address__, __meta_kubernetes_pod_annotation_prometheus_io_port ]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: labeldrop
        regex: statefulset_kubernetes_io_pod_name # This is duplicate because we already have the pod_name
      - source_labels: [ __meta_kubernetes_namespace ]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [ __meta_kubernetes_pod_name ]
        action: replace
        target_label: kubernetes_pod_name

Accordingly I set the annotations on the to be scraped pods:

metadata:
  annotations:
    checksum/config: "8755a7a06e8302a6861c536738da919828761d7c6a794b2c52c920b4ddae163e"
    cni.projectcalico.org/podIP: "172.22.130.165/32"
    kubernetes.io/psp: "prometheus"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "9090"
    prometheus.io/scrape: "true"

Now for some reason all of these targets appear twice (under different ports) in the scrape targets as shown in the Prometheus UI:

Why is this the case? Why does it appear under port 80? The target with port 9090 shows that the relabel config where it reads the port from the annotation works.

Additional information:

the prometheus pod is part of a stateful set
it defines just one (unnamed) port 9090
it has an init container that doesn't expose a port
the screenshot oviously only shows scrape targets from the kubernetes-pods job
this is just prometheus (v2.22.0) and is not managed by the prometheus operator

Do you have a screenshot of the service discovery page of these two targets? — weibeld
@weibeld I can provide them, unfortunately due to the sheer amount of labels it's hard to get them onto one picture. Are you looking for something particular? The pod prometheus-0 appears twice in the service discovery. Once with instance = "ip:80" and the second with instance = "ip:9090" — kentor

weibeld weibeld · Accepted Answer · 2020-10-26T11:01:43

The pod service discovery role of Prometheus creates a target for each container of your Pod, including init containers. You have an init container in your Pod, so this results in two targets.

If a container does not declare a port (like your init container), then a port-free target is created. That means, the regex in your replace rule that sets __address__ does not match and Prometheus then uses port 80 by default for this target.

To drop the init container target, try adding the following rule:

- source_labels: [ __meta_kubernetes_pod_container_init ]
  regex: true
  action: drop

Duplicate scrape targets on different ports

1 Answers