Prometheus/PromQL subtract two gauge metrics

Question

I have this gauge metric "metric_awesome" from two different instances. What i want to do, is subtract instance one from instance two like so

metric_awesome{instance="one"} - metric_awesome{instance="two"}

Unfortunately the result set is empty. Has anyone experienced this?

Answer 1

The issue here is that the labels don't match. What you want is:

metric_awesome{instance="one"} - ignoring(instance) metric_awesome{instance="two"}

Answer 2

If anyone searches this wanting to do this for a one-to-many subtraction, have a look at group_right additionally to what was written before.

metric_awesome{instance="one"} - ignoring(instance) group_right metric_awesome{job="compare-instances"}

See also Prometheus dokumentation