How do I combine series based on multiple variables in Grafana

Question

On my Graphite database, I have series that look like this :

server.[server-name].app.[app-name].memory.used
server.[server-name].app.[app-name].memory.committed

For example :

server.backend.app.apiserver.memory.used
server.backend.app.apiserver.memory.committed
server.backend.app.database.memory.used
server.backend.app.database.memory.committed
server.frontend.app.webserver.memory.used
server.frontend.app.webserver.memory.committed

Using Graphana, I would like to plot a single graph where I show the amount of free memory on multiple application based on template values that I may filter by server name(s) or application name(s). I also want the time series to be aliased by the server name and application name.

For example, in the example above, I'd like to see 3 series on a single graph :

Series 1, free memory for backend.apiserver
Series 2, free memory for backend.database
Series 3, free memory for frontend.webserver

I'm able to do this by selecting a single server and a single application in a template using a function such as :

Series A(hidden ) : server.$server.app.$app.memory.used
Series B(visible) : diffSeries(server.$server.app.$app.memory.committed), #A)

However, if want the graph to show multiple servers and multiple applications at once, Series B just becomes a subtraction of all the values in the A series.

An easy answer would be to send the "free" memory data explicitly to Graphite as a third data point on every application, but I was wondering if the Grafana dashboard had a function that could accomplish the same results without tampering with the existing data collection scripts I have.

You could try using the groupByNode function to do your diffSeries function; Eg: groupByNode(server.$server.app.$app.memory.{committed,used}, 1, "diffSeries") The second argument is zero-indexed. So we're telling it to group by $server, and pass each series from that grouping to the diffSeries function. But doing this seems to lose the $app part... so I'm unsure how to label it how you want. — Chris
Thanks for the suggestion. I changed the groupByNode to use series 3 instead of 1 so that each app has it's own report. It works well as long as I pin down the dashboard to a single $server. Not perfect, but better than what I had so far. — Steve McDuff

kwarunek kwarunek · Accepted Answer · 2020-06-04T00:49:28

Depending of graphite version it could be done with reduceSeries function (doc has similar example).

reduceSeries(
    mapSeries($server.app.$app.memory.*, 0, 2),
    "diffSeries", 4,"commited", "used"
)

To be fair, I haven't tested with grafana's templating/variable mechanism.

How do I combine series based on multiple variables in Grafana

1 Answers