How to plot average per day from Prometheus

Question

I have to create a visualization from Prometheus metrics. I have a counter metrics http_request_counter, I wanted to show a summary of total requests served in a day. This is how I did when we had graphite as our data source.

alias(summarize(sumSeries(consolidateBy(nonNegativeDerivative(http_request_counter.count), 'sum')), '1d', 'sum', false), 'TPS per day')

I saw few documentations and tried with increase(http_requests_total[24h]) which plotted the graph with t-24h values.

Can someone help me find an equivalent of summarize function in prometheus please?

Answer 1

You need sum(increase(http_requests_total[24h])). It returns the summary number of requests during the last 24 hours for every point on the graph. By default Grafana queries many points from Prometheus in order to draw smooth graph for the current horizontal resolution of the graph. The number of points on the graph can be adjusted with the min step option when editing the graph in Grafana. Another option is to use subqueries with the desired step for the outer query:

last_over_time(sum(increase(http_requests_total[24h]))[1d:1d])

It will return continous line with one-day steps. These steps will be shifted by one day in the past, e.g. the step for the current day will contain the number of requests made on the previous day. This shift can be removed with negative offset:

last_over_time(sum(increase(http_requests_total[1d] offset -1d))[1d:1d])

Unfortunately, this query doesn't work out of the box in Prometheus, since it doesn't accept negative offsets by default. They must be enabled with --enable-feature=promql-negative-offset command-line flag when starting Prometheus. Fortunately, negative offsets are supported by default in MetricsQL.

Answer 2

I am still not sure whether you want total requests per day or average req/s per day, here is expression for the latter:

avg_over_time(
  sum(rate(http_request_counter[5m])) by (method, some_other_label)[1d]
)

Since it is a counter, you need to apply rate to get a per-second vector, then use sum if needed to aggregate across multiple series on some labels, and at last average over one day.