Kamon, Statsd, Grafana Disk Space

Question

I recently setup Kamon, Statsd, Graphite, Grafana to monitor my akka application. I used the Docker container for Statsd, Graphite and Grafana that most tutorials suggest and it all worked fine until recently when I noticed the Grafana dashboard was starting to lag and become unresponsive. Its now become unresponsive and the reason is the machine hosting the Docker container has run out of space...

85790556    /var/lib/docker
85786096    /var/lib/docker/aufs
42922104    /var/lib/docker/aufs/diff
42863828    /var/lib/docker/aufs/mnt
42863668    /var/lib/docker/aufs/mnt/1daed028a6ca7041dc95257b27328324936f548d2cfc589d6e3c94ac3d8cc63b
41878192    /var/lib/docker/aufs/mnt/1daed028a6ca7041dc95257b27328324936f548d2cfc589d6e3c94ac3d8cc63b/opt
41793244    /var/lib/docker/aufs/diff/1daed028a6ca7041dc95257b27328324936f548d2cfc589d6e3c94ac3d8cc63b
41754416    /var/lib/docker/aufs/mnt/1daed028a6ca7041dc95257b27328324936f548d2cfc589d6e3c94ac3d8cc63b/opt/graphite

This is likely from the monitoring stats building up. So I've two questions...

How do I safely clean up space in this container to get it running again.
Can I configure statsd or graphite to periodically purge data?

Thanks!

vdebergue vdebergue · Accepted Answer · 2016-11-24T22:10:57

A better solution would be to configure carbon to manage the retention of your data. You can define several intervals to keep your data and at which resolution.

For instance in storage-schemas.conf:

retentions = 15s:7d,1m:21d,15m:5y

This retention will mean that the metrics will be kept for 7 days with a resolution of 15 seconds. Then carbon will aggregate the older data with a precision of 1 min and keep them for 21 days. Then keep the data for 5 years with a resolution of 15 min.

Depending on your need and your storage capacity, you can configure several retention policies.

In storage-aggregation.conf you can define how carbon should aggregate your metrics when changing the resolution: take the min, max, avg, sum ...

More information in carbon documentation

Kamon, Statsd, Grafana Disk Space

2 Answers