Creating one graph per node in Graphite and statsd

Question

I am using graphite with statsd to collect various pieces of data from an application I have clustered in ec2. Among other things I have statistics for messages delivered by by application ond those that failed delivery. Because the number of consumers of messages of my application is somewhat fluid (changes every few days) I would like to be able to dynamicaaly generate graphs for these two stats per consumer. I post messages to statsd/graphite with the consumer id that is generated by the system as part of the stat:

stats.myapp.consumers.<consumerId>.messagesDelivered
stats.myapp.consumers.<consumerId>.messagesFailed

I currently generate two graphs using the wildcard for the consumerId:

stats.myapp.consumers.*.messagesDelivered
stats.myapp.consumers.*.messagesFailed

This is ok for a few consumers but I currently have >300 and it is growing rapidly. What I would like to be able to do is pass the above targets (or similar) and have graphite return a graph for each unique id. (I say graph but I am actually getting json results) I am currently using a personally modified version of the giraffe dashboard but am open to other suggestions.

Is this something that is even possible? (I suppose I could write some custom code to retrieve each id then generate the requests for each one each time. But I would really like to keep my customizations minimal if possible.)

I'm sure it's likely I will need to provide more details. Please let me know what you would need to know or if I am possibly just off-base with my request.

More details:

So, the end result of what I am hoping for is not to have one graph with, for lack of a better term, 300 lines each representing a node. I would like to have one graph for each node, each with two lines. One for delivered messages and one for failed messages.

When I attempted the solution from @Matt Self I ended-up with one graph with 300 lines on it. (Or, maybe I misinterpreted the suggestion?)

Matt Self Matt Self · Accepted Answer · 2013-06-26T14:32:16

If I understand you correctly, I believe you can use groupByNode to do what you want:

groupByNode(stats.myapp.consumers.*.messagesDelivered,3,"sumSeries")

The 3 is used to indicate the unique id part of the bucket (e.g. stats=0,myapp=1,consumers=2,uniqueid=3,etc). And I used the sumSeries as the callback function in the example above, which it seems like might work for you, but you'll have to ensure you choose a function that makes sense for any series data you get per a unique id.

groupByNode used in this way (with the wildcard for the unique id) will return separate targets with datapoints for (grouped-by) each uniqueid.

Update based on comments:

While the above will create one rendered graph image with many graphs/lines based on a single input "target", the OP wants to create multiple independent graph images based on one input:

I don't know of a way render multiple graphs from the graphite url api (i.e. using graphite alone). Perhaps there is something in the dashboard that will accomplish it.

That said, you could do this with some simple html + javascript. Something like this might work (but you might have to tweak it):

<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8" />
    <title>Multi-graph</title>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js" type="text/javascript"></script>

    <script type="text/javascript">
        var graphite_render_url = "http://some.graphite.host.blah.com/render/";
        var mainTarget = "stats.myapp.consumers.*.messagesDelivered";
        var mainFunc = "sumSeries";
        var url = graphite_render_url + "?target=groupByNode(" + mainTarget + ",3,\"" + mainFunc + "\")";
        url = url + "&format=json&jsonp=?";

        $.getJSON(url, function(data){
        for (var i = data.length - 1; i >= 0; i--) {
            console.log(data[i]);
            var imgSrc = graphite_render_url + "?target=";
            var imgActual = mainTarget.replace('*',data[i].target);
            $('#imagesList').append('<img src="' + imgSrc + imgActual + '" />');
        };
    });
    </script>

</head>
<body>
<div id="imagesList"></div>
</body>
</html>

The above would insert an image for every unique returned target. This might not be exactly what you are looking for, but might get you headed down a working path.

Creating one graph per node in Graphite and statsd

1 Answers