Best retention practice using Graphite

Question

I have been a happy user of Graphite+Grafana for a few months now and I have been advocating it around my firm.

My approach has been to measure data of interest and collect them into 1-minute or 5-minute buckets and send that information to Graphite. I was recently contacted by a group that processes quotes (billions a day!) and their approach has been to create a log line each time their applications process 1 million quotes. The problem is that the interval between 2 log lines can be highly erratic from 1 second to a few hours.

The dilemma is then: should I set my retention policy to a 1-second bucket so that I can see all measurements associated with spikes or should I use say a 1-minute bucket so that the number of data points to be saved and later on queried is much more manageable. FYI, when I set it to 1-second, showing the data for 8 or 10 charts, for a few days was bringing the system (or at least my browser) to a crawl because of the numbers of data points (mostly NULL) being pushed around from Graphite to Grafana

Here's my retention policy: 1s:10d,1m:36d,5m:180d

Alternatively, is there a way to configure Grafana+Graphite to only retrieve non-NULL data points?

What do you recommend?

Answer 1

You can always specify a lower retention period for 1s metrics so when you show a longer range Graphite will send you only the more coarse level.

For example, you can specify: 1s:2d, 1m:7d, 5m:180d

This way, if you show a range more than 2 days in the past you will get 1m resolution (and so on), which won't make your browser crawl, while you will still be able to inspect spikes in the last 2 days.