statsd architecture for a distributed system

Question

I am studying to use the graphite - statsd - collectd stack to monitor a a distributed system.

I have tested the components (graphite-web, carbon, whisper, statsd, collectd and grafana) in a local instance.

However I'm confused about how I should distributed these components in a distributed system: - A monitor node with graphite-web (and grafana), carbon and whisper. - In every worker node: statsd and collectd sending data to the carbon backend in the remote monitor node.

Is it right this scheme? What I should configure statsd and collectd to get an acceptable network ussage (tcp/udp, packets per second...)?

Answer 1

Assuming you have a relatively light workload, having a node that manages graphite-web, grafana, and carbon (which itself manages the whisper database) should be fine.

Then you should have a separate node for your statsd. Each of your machines/applications should have statsd client code that sends your metrics to this statsd node. This statsd node should then forward these metrics onto your carbon node.

For larger workloads that stress a single node, you'll need to either scale vertically (get more powerful node to host your carbons/statsd instances), or start clustering those services.

Carbon clusters tend to use some kind of relay that you send to that manages forwarding those metrics to the cluster (usually using consistent hashing). You could use a similar setup to consistently hash metrics to a cluster of statsd servers.