Deleted/Empty Graphite Whisper Files Automatically Re-Generating

Question

I am trying to delete some old graphite test whisper metrics without any success. I can delete the metrics by removing the files. (See: How to cleanup the graphite whisper's data? ) But, within a few seconds of blowing away the files they regenerate (they are empty of metrics and stay that way since nothing is creating new metrics in those files). I've tried stopping carbon (carbon-cache.py stop) before deleting the files, but when I restart carbon (carbon-cache.py --debug start &) they just come back.

How do I permanently delete these files/metics so they never come back?

Dave Strock Dave Strock · Accepted Answer · 2013-11-15T22:44:57

By default, Statsd will continue to send 0 for counters it hasn't received in the previous flush period. This causes carbon to recreate the file.

Lets say we want to delete a counter called 'bad_metrics.sent' from Statsd. You can use the Statsd admin interface running on port 8126 by default:

$ telnet <server-ip> 8126
Trying <server-ip>...
Connected to <server-name>.
Escape character is '^]'.

Use 'help' to get a list of commands:

help
Commands: stats, counters, timers, gauges, delcounters, deltimers, delgauges, quit

You can use 'counters' to see a list of all counters:

counters
{ 'statsd.bad_lines_seen': 0,
  'statsd.packets_received': 0,
  'bad_metrics.sent': 0 }
END

Its the 'delcounters', 'deltimers', and 'delgauges' commands that remove metrics from statsd:

delcounters bad_metrics.sent
deleted: bad_metrics.sent
END

After removing the metric from Statsd, you can remove the whisper file associated with it. In this example case, that would be:

/opt/graphite/storage/whisper/bad_metrics/sent.wsp

or (in Ubuntu):

/var/lib/graphite/whisper/bad_metrics/sent.wsp

Deleted/Empty Graphite Whisper Files Automatically Re-Generating

4 Answers