Prometheus query.lookback-delta recommended value

3
votes

I read and understood the concept of staleness involved with Prometheus 2.0 here

Within the exporter I'm developing, metrics are pushed by the remote devices as a gRPC stream so I create metrics on the fly using prometheus.NewMetricWithTimestamp and implement the Collector interface.

Whenever the remote device stops emitting metric(s), these are not exposed anymore in the Prom client HTTP endpoint. However, the default value for query.lookback-delta flag (5 min), makes Prometheus marking the metric as stale for 5 minutes. So when sending the PromQL query, we still see the last value scraped.

I did set the flag to 30s, so from a dashboard perspective, the end user doesn't have to wait 5 minutes to see the metrics disappearing.

Therefore, I would like to know whether 30s is reasonable value and if not what are the implications ?

Thanks in advance.

1
prometheus

1 Answers

1
votes

As the doc you linked said, lookback-delta is mainly to align different time series for aggr, and you can adjust this based on your scrape interval, take an example:

if scrape interval is 10s, then 30s is reasonable setting, suitable for align and avoid seldom scrape error.