Prometheus - query to detect the number of pods that existed in the last 24 hours

Question

I'm still getting to grips with PromQL. I wrote this query in an attempt to detect the number of kubernetes pods that existed in the last 24 hours within a given namespace.

My process here was:

• Get the metric filtered to the relevant name-spaces (any airflow ones).
• Get that metric over 24 hours.
  • Each pod will just have lots of duplicates of the same creation time here.
• Use increase() to get the range vectors for each pod back into instant vectors. The value will always be 0 as the creation time does not increase.
• Now that we have 1 value per pod, use count() to see how many existed in that time frame.

count(increase(kube_pod_created{namespace=~".*-airflow"}[1d]))

Can anyone that knows prometheus well tell me if this logic follows? Since it isn't a normal database/etc I'm having trouble working out how to validate this query. It "looks" like it probably does the right thing when expanded out to a day though.

Answer 1

I'd recommend substituting increase() with count_over_time(), since increase may miss short-living pods with lifetime smaller than 2x scrape interval. The following query should return the total number of pods seen during the last 24 hours:

count(count_over_time(kube_pod_created{namespace=~".*airflow"}[24h]))