How to alert in Seyren with Graphite if transactions in last 60 minutes are less than x?

Question

I'm using Graphite+Statsd (with Python client) to collect custom metrics from a webapp: a counter for successful transactions. Let's say the counter is stats.transactions.count, that also has a rate/per/second metric available at stats.transactions.rate.

I've also setup Seyren as a monitor+alert system and successfully pulled metrics from Graphite. Now I want to setup an alert in Seyren if the number of successful transactions in the last 60 minutes is less than a certain minimum.

Which metric and Graphite function should I use? I tried with summarize(metric, '1h') but this gives me an alert each hour when Graphite starts aggregating the metric for the starting hour.

Note that Seyren also allows to specify Graphite's from and until parameters, if this helps.

Answer 1

I contributed the Seyren code to support from/until in order to handle this exact situation.

The following configuration should raise a warning if the count for the last hour drops below 50, and an error if it drops below 25.

• Target: summarize(nonNegativeDerivative(stats.transactions.count),"1h","sum",true)
• From: -1h
• To: [blank]
• Warn: 50 (soft minimum)
• Error: 25 (hard minimum)

Note this will run every minute, so the "last hour" is a sliding scale. Also note that the third boolean parameter true for the summarize function is telling it to align its 1h bucket to the From, meaning you get a full 1-hour bucket starting 1 hour ago, rather than accidentally getting a half bucket. (Newer versions of Graphite may do this automatically.)

Your mileage may vary. I have had problems with this approach when the counter gets set back to 0 on a server restart. But in my case I'm using dropwizard metrics + graphite, not statsd + graphite, so you may not have that problem.

Please let me know if this approach works for you!