prometheus - total metric count and prometheus_tsdb_head_series results differ

1
votes

count({__name__=~".+"}) query shows only 9 but prometheus_tsdb_head_series shows 837 count, any idea why is that difference? are both queries not similar?

here is the scrape config:

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus-1'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']
      labels:
        env: local
    metric_relabel_configs:
    - source_labels: [__name__]
      regex: (prometheus_tsdb_head_series)
      action: keep


  - job_name: 'node_exporter-1'
    static_configs:
    - targets: ['localhost:9100']
      labels:
        env: local
    metric_relabel_configs:
    - source_labels: [__name__]
      regex: (?i)(metric1|metric2|metric3)
      action: keep
2
prometheus

2 Answers

7
votes

prometheus_tsdb_head_series covers every series that has existed in the last 1-3 hours, count({__name__=~".+"}) covers series that are not stale in the past 5 minutes.

Given that config, I'd guess that these other series are from before you added the metric_relabel_configs.

0
votes

I found the problem, there was another prometheus process running on localhost:9090, second prometheus process with above config was scraping this different prometheus instance thats the reason there was difference in between these 2 queries, after correcting the port in scrape config, I can see both queries returning same result.