Is Graphite usable with very occasional counters?

Question

Using statsd, configured with flushInterval: 1000 and communicating with graphite's carbon-cache. I wish I can see very occasional counters.

I have the following configuration for carbon:

storage-schemas.conf:

[carbon]
pattern = ^carbon\.
retentions = 60:90d

[default_30s_for_1day]
pattern = .*
retentions = 30s:1d

Sending a unique counter that way:

$ echo "foobar:1|c" > /dev/udp/127.0.0.1/8125

I can see the packet received by statsd:

9 Jul 14:43:05 - DEBUG: foobar:1|c

as well as the data sent to carbon-cache (tcpdump extract):

stats.foobar 1 1404909785

In graphite, looking at data for "foobar" I can see that something happened at that moment (thin line, see red circle in picture), but the result is always on "0":

enter image description here

Am I missing something?

If there is much more frequent results, then I can see numbers that looks correct.

Is there a minimum amount of stats to be sent to be taken into account? Is it configurable?

Note: Maybe for such occasional data StatsD / Graphite is not worth but since there are other very frequent data collected for the same project, those will be used anyway and hope one unique solution can be used, even for rare counters.

Patrick Allaert Patrick Allaert · Accepted Answer · 2014-07-09T16:18:50

The issue in my setup was that the flushInterval (1 sec.) was lower than my smallest retention (30 sec.)

Every (my case: 30 sec.), carbon-cache stores the result, however if, e.g., at second "10" the count "1" is sent for my foobar key, statsd will send the count "0" every second (every flushInterval to be more precise) after. This means that at second "30", the last value seen for foobar is "0". The only chance that the value "1" would be taken into account is if that was sent at the very latest moment, just before second "30".

One can blame me not to use statsd deleteIdleStats or deleteCounters settings to not send "0". But doing so, having two times the counter set to "1" in the same 30 sec. slot would miss the first counter, resulting in a count of "1" being registered instead of "2".

The real solution is to align statsd's flushInterval with the minimum retention of carbon:

statsd config.js:

flushInterval: 10000

carbon's storage-schemas.conf:

retentions = 10s:1d,1m:30d,5m:90d

Is Graphite usable with very occasional counters?

1 Answers