Prometheus - same metric with different labels is overridden before scraping

1
votes

I'm consuming various gauge metrics from Kafka to Prometheus using custom python app. The metrics needs to be consumed in more and less the same time (milliseconds). I'm able to successfully scrape only the last metric from below example as the first three are immediately overwritten.

my_metric{labelA = "aaa", labelB = "111"} 8
my_metric{labelA = "aaa", labelB = "222"} 12
my_metric{labelA = "bbb", labelB = "111"} 7
**my_metric{labelA = "bbb", labelB = "222"} 15**

I can get all four metrics getting scraped by setting them a unique metrics name e.g.:

my_metric_aaa_111{labelA = "aaa", labelB = "111"} 8

but this does not seem to be the best practice plus working with such metrics is very difficult in general later on in Grafana.

I can also push the metrics serially to be scraped and lower the interval of scraping in Prometheus config, but this is against the whole idea of the solution.

Apart from suggestions I'm completely unaware - is it possible to keep for Prometheus the same metric to be scraped where only label values differ? The only discussion I found on this is here with no answer: https://github.com/prometheus/pushgateway/issues/65.

If the above is not possible, can I somehow merge/join/combine metrics name later on in Prometheus/Grafana to be able to work with them based on their labels? Meaning remove the unnecessary ending _aaa_111 in the example above to work back with everything as one metrics?

3
pythongrafanaprometheus
What do you mean by 'the first three are immediately overwritten` ? a metric is uniquely identified by its name and the set of labels. In your example, they are four different metrics and will be parsed as such. - Michael Doubez
Thanks @Michael for pointing me to the right direction. If for Prometheus all four metrics are considered to be unique I will need to look into the python producer app to find out what is wrong in there. - Jonas S
FYI it's scrape (and scraping, scraped, scraper) not scrap. 'To scrap' means to throw away like rubbish :-( - balmy

3 Answers

2
votes

You can change the Pushgateway job name, using a combination of the labels, or some unique value. So prometheus could scrape all your metrics, and they aren't overwriten by others.

In your case, if you export: my_metric{labelA = "aaa", labelB = "111"} 8 to a job called, some_job_aaa_111.

You can test pushing some metrics manually: echo "my_metric{labelA = \"aaa\", labelB = \"111\"} 8" | curl --data-binary @- http://localhost:9091/metrics/job/some_job_aaa_111

In Prometheus pushgateway you would see the next:

Prometheus pushgateway jobs

In Prometheus, the pushgateway job, becomes in a label that you can easily ignore, like the next output of Prometheus:

Prometheus jobs

0
votes

Another Solution: Make the scraping action be more frequently repetitive using scrape_interval attribute on prometheus.yml . If you based on minutes ,convert it to seconds or milliseconds to avoid race conditions.

- job_name: 'pushgateway'
  honor_labels: true
  scrape_interval: 50ms
  metrics_path: /metrics
  static_configs:
    - targets:
      - localhost:9091
0
votes

You could also have different grouping keys when pushing to the push-gateway (and will look better than having the same prefix and different suffixes for job names)