Simple cumulative increase in Prometheus

Question

I have an application that increments a Prometheus counter when it receives a particular HTTP request. The application runs in Kubernetes, has multiple instances and redeploys multiple times a day. Using the query http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"} produces a graph displaying cumulative request counts per instance as is expected.

I would like to create a Grafana graph that shows the cumulative frequency of requests received over the last 7 days.

My first thought was use increase(...[7d]) in order to account for any metrics starting outside of the 7 day window (like in the image shown) and then sum those values.

I've come to the realisation that sum(increase(http_requests_total{method="POST",path="/resource/aaa",statusClass="2XX"}[7d])) does in fact give the correct answer for points in time. However, resulting graph isn't quite what was asked for because the component increase(...) values increase/decrease along the week.

How would I go about creating a graph that shows the cumulative sum of the increase in these metrics over the passed 7 days? For example, given the simplified following data

| Day | # Requests |
|-----|------------|
| 1   | 10         |
| 2   | 5          |
| 3   | 15         |
| 4   | 10         |
| 5   | 20         |
| 6   | 5          |
| 7   | 5          |
| 8   | 10         |

If I was to view a graph of day 2 to day 8 I would like the graph to render a line as follows,

| Day | Cumulative Requests |
|-----|---------------------|
| d0  | 0                   |
| d1  | 5                   |
| d2  | 20                  |
| d3  | 30                  |
| d4  | 50                  |
| d5  | 55                  |
| d6  | 60                  |
| d7  | 70                  |

Where d0 represents the initial value in the graph

Thanks

Did you ever figure out a way to do this? I am attempting to do the same and coming up empty — Zane DeGraffenried

nezzzzzz nezzzzzz · Accepted Answer · 2021-09-07T11:59:06

If I understood your question's idea correctly, I think I managed to create such graph with a query like this

sum(max_over_time(counterName{someLabel="desiredlabelValue"}[7d]))

A graph produced by it looks like the blue one:

The reasons why the future part of the graph decreases are both because the future processing hasn't obviously yet happened and because the more-than-7-days-old processing slides out of the moving 7-day inspection window.

Simple cumulative increase in Prometheus

1 Answers