InfluxDb high high cardinality data

Question

I'm inserting data into the influxDb using batch points via Java API (used http API under the hood) after some time the exception is raised.

java.lang.RuntimeException: {"error":"partial write: max-values-per-tag limit exceeded (100010/100000):

According to the Influx docs - docs this parameter prevent high cardinality data from being written before it can be fixed into the Influx.

I can set it to 0 to remove Exception. But I don't clear understand what is "high cardinality data". What's wrong to have "high cardinality data" to be inserted into InfluxDb. I'm going to insert millions unique values and need them to be indexed. Do I need to review my data design ?

Answer 1

They are using in memory index for "tags", the more different tags values you have (the higher aardinality of the data) the more memory influx requires.

https://github.com/influxdata/influxdb/issues/7151

Answer 2

InfluxDB may require high amounts of memory for high cardinality data (~10KB of memory per time series) and memory requirements may grow exponentially with the number of unique time series. See these official docs for details.

There are other time series databases exist, which require lower amounts of RAM for high cardinality data. For instance, this benchmark compares memory usage and performance of InfluxDB and VictoriaMetrics - an alternative TSDB, which understands Influx line protocol.