ElasticSearch Cardinality issue

0
votes

Cardinality Aggregation calculates an approximate count of distinct values. But why even for an index stored in a single shard, it is showing incorrect value?


    GET /jobs/_settings

    {
      "jobs": {
        "settings": {
          "index": {
            "number_of_shards": "1",
    ...


    position_id is long

    GET /jobs/_search
    {
      "size": 0,
      "aggs": {
        "count_position_id": {
          "value_count": {
            "field": "position_id"
          }
        },
        "unique_position_id": {
          "cardinality": {
            "field": "position_id",
            "precision_threshold": 40000
          }
        }
      }
    }

    {
      "took": 44,
      "timed_out": false,
      "_shards": {
        "total": 1,
        "successful": 1,
        "failed": 0
      },
      "hits": {
        "total": 52836,
        "max_score": 0,
        "hits": []
      },
      "aggregations": {
        "unique_position_id": {
          "value": 52930
        },
        "count_position_id": {
          "value": 52836
        }
      }
    }
1
elasticsearchdistinctcardinality

1 Answers

1
votes

Its more to do with algorithm used to calculate cardinality than single shard in picture.

ES cardinality agg works using HLL ( hyperloglog) which is approximate counting algorithm ( It relies on observation on binary representation of hash to approximate unique values count )

You can control precision by increasing precision_threshold .So by definition this is "approximate count" - not really incorrect.