Unknown number of metrics received with statsd and graphite

Question

I'm trying to gather some data on the performance of graphite and the carbon daemon. Luckily for me the carbon daemon reports to graphite every 60 seconds with some stats on its workings such as the number of metrics received.

I'm using statsd to aggregate stats and flush them to the carbon daemon every second, but noticed some weird behavior when setting up to show the number of metrics received in a certain time interval. I'm using grafana to connect to my Graphite instance and pull data out of it. Whenever statsd is not running and I inspect the number of metrics received it remains 0, which is not weird considering nothing is sending it anything. However, when I start statsd the number quickly rises to about 800/900 per minute without me sending any stats to it yet as can be seen in this image: Metrics received when statsd is running

I'm at a loss as to where these metrics are coming from and why they happen at a rate of 15 per second. Also, the CPU increases the load by about 10% whenever I start statsd. What I did notice is that when I increase the flush interval of statsd the number of metrics received decreases.

This is my configuration file of statsd:

{
  graphitePort: 2003,
  graphiteHost: "127.0.0.1",
  port: 8125,
  backends: ["./backends/graphite"],
  flushInterval: 1000, // Don't increase this past the lowest retention schema of graphite
  prefixStats: "test",
  graphite: {
    legacyNamespace: false
  }
}

And here's my storage schema for graphite:

[carbon]
pattern = ^carbon\.
retentions = 60s:90d,300s:365d

[stats]
pattern = ^stats\..*
retentions = 1:2160,10:2160,60:10080,600:262974

[system]
pattern = ^system\..*
retentions = 10:2160,60:10080,600:262974

dukebody dukebody · Accepted Answer · 2015-04-17T07:36:09

I believe statsd sends its own performance metrics to Graphite even if you're not sending any metric of your own to statsd. For example, I can find stats.statsd.processing_time, stats.statsd.graphiteStats.*...

It's the same as with Carbon, where you get a metric of the number of metrics received even if you're not sending anything. Think of them as internal metrics.

Check out https://github.com/etsy/statsd/blob/1590bcf56ea1a3ac167f62fba3d599b65582d5ea/docs/admin_interface.md and https://github.com/etsy/statsd/blob/917a3e4276daf2587c30fcfcce65b0066e776626/backends/graphite.js

Unknown number of metrics received with statsd and graphite

2 Answers