I have a use-case of monitoring that I'm not entirely sure if it's a good match for Prometheus or not, and I wanted to ask for opinions before I delve deeper.
The numbers of what I'm going to store:
Only 1 metric. That metric has 1 label with 1,000,000 to 2,000,000 distinct values. The values are gauges (but does it make a difference if they are counters?) Sample rate is once every 5 minutes. Retaining data for 180 days.
Estimated storage size if I have 1 million distinct label values:
(According to formula in Prometheus' documentation: retention_time_seconds * ingested_samples_per_second * bytes_per_sample)
(24*60)/5=288 5-minute intervals in a day.
(180*288) * (1,000,000) * 2 = 103,680,000,000 ~= 100GB
samples/label-value label-value-count bytes/sample
So I assume 100-200GB will be required.
Is this estimation correct?
I read in multiple places about avoiding high-cardinality labels, and I would like to ask about this. Considering I will be looking at one time-series at a time Is the problem with high-cardinality labels? Or having a high number of time-series? As each label value produces another time-series? I also read in multiple places that Prometheus can handle millions of time-series at once, so even if I have 1 label with one million distinct values, I should be fine in terms of time-series count, do I have to worry about the labels having high cardinality in this case? I'm aware that it depends on the strength of the server, but assuming average capacity, I would like to know if Prometheus' implementation has a problem handling this case efficiently.
And also, if it's a matter of time-series count, am I correct in assuming that it will not make a significant difference between the following options?
- 1 metric with 1 label of 1,000,000 distinct label values.
- 10 metrics each with 1 label of 100,000 distinct label values.
- X metrics each with 1 label of Y distinct label values. where X * Y = 1,000,000
Thanks for the help!
