Creating prometheus metrics with variable label names

Question

I am trying to take some dictionaries that get passed to this program, and export prometheus metrics.

The dictionaries are non nested

{'node-name': '0/0/CPU0', 'stack-seg-size': 136, 'text-seg-size': 108, 'shared-mem': 12748, 'process-id': 4086, 'physical-mem': 0, 'malloc-size': 1238, 'name': 'l2snoop', 'pid': 4086, 'data-seg-size': 533588, 'dyn-limit': 307200, 'jid': 251}

However the number of keys or labels in prometheus metrics might vary for dictionary to dictionary.

I want to end up with something like this:

METRIC-NAME{node-name='0/0/CPU0', stack-seg-size='136', text-seg-size= '108', shared-mem='12748', process-id='4086', physical-mem='0', malloc-size='1238', name='l2snoop', pid='4086', data-seg-size='533588', dyn-limit= '307200', jid='251'} ''

So I can use grafana to graph label values http://docs.grafana.org/features/datasources/prometheus/#query-variable

I have the following code:

        counters = CounterMetricFamily('{}'.format(metric_name), 'Metrics for {}'.format(metric_name))
        for metric in traverse_tree(data): #This returns a list of the dictionaries 
            metric = ['{}={}'.format(k,v) for k,v in metric.items()]
            counters.add_metric(metric, "")
        yield counters

No errors are thrown but when I go to the metric page there is no metrics, I think due to the face the label names are not defined before.

brian-brazil brian-brazil · Accepted Answer · 2018-02-23T07:25:15

Having label values that vary like that will result in a high cardinality metric which will cause performance problems in Prometheus. You also can't do math on label values.

What you want to do is create a differently named metric per value, These are also all Gauges, bar node-name which is not a metric.

Creating prometheus metrics with variable label names

1 Answers