prometheus custom collector with "cached" metrics

Question

i want to have a prometheus collector that can be queried via http on port 9100 which is delivering a metric. the curl should always just provide the latest metric i am creating with my function here:

https://github.com/unnuetz/custom-prometheus-collector/blob/master/check.py#L5-L11

my problem now is, that after each cycle of 100secondes i added here: https://github.com/unnuetz/custom-prometheus-collector/blob/master/main.py#L20-L23 the next metric i am creating is added, but i would like to have a "reset" after each call of checker(), and not that the individual metrics i created are always just added to my lets say registry. is there a way, to remove all metrics which were added before? the important thing ist, that between the 100seconds the start_http_server(9100, addr='0.0.0.0') is still availabe and delivering the latest metric data it have from the last checker() run. in my demo file i always want to have just one metric with an random generated label name.

Michael Doubez Michael Doubez · Accepted Answer · 2020-05-10T15:59:34

Instead of using using builtin Gauge, you can use a custom collector.

Note that it won't save you in Prometheus:

it will create a new metrics each time prometheus scrapes your application
the previous metric(s) will still be visible in Prometheus until it is marked as stale (~5 min).

If using using a custom collector is not an option because you need to show some code using global metrics. You can inject a reset() function similar to the remove() function using a bound method.

import CollectorRegistry from prometheus_client.registry

def _resetCollector(self):
    with self._lock:
        # reset whatever name / value you want
        pass

CollectorRegistry.reset = _resetCollector


# wherever you want to reset it
import REGISTRY from prometheus_client.registry
REGISTRY.reset()

Of course, it will break whenever the implementation of CollectorRegistry change.

prometheus custom collector with "cached" metrics

1 Answers