Elasticsearch returns 0.0 for metrics sum aggregation

Question

Elasticsearch returns 0.0 for metrics sum aggregation. Expected output will be some of metric probe_http_duration_seconds.

Elasticsearch version: 7.1.1

Query used for aggregation:

GET some_metric/_search
{
"query": {
  "bool": {
    "must": [
      {
        "range": { "time": { "gte" : "now-1m", "lt": "now" } }
      },
      {
        "match": {"name": "probe_http_duration_seconds"}
      },
      {
        "match": {"labels.instance": "some-instance"}
      }
    ]
  }
  },
  "aggs" : {
        "sum_is" : { "sum": { "field" : "value" } }
    }
}

The above query returns for documents followed by:

"aggregations" : {
  "sum_is" : {
    "value" : 0.0
  }

Each document in the index looks like:

  {
    "_index" : "some_metric-2019.12.03-000004",
    "_type" : "_doc",
    "_id" : "_wCjz24Bk6FPpmW1lC31",
    "_score" : 5.3475914,
    "_source" : {
      "name" : "probe_http_duration_seconds",
      "time" : 1575441630181,
      "value" : 0,
      "labels" : {
        "__name__" : "probe_http_duration_seconds",
        "app" : "some-events",
        "i" : "some_metric",
        "instance" : "some-instance",
        "job" : "someproject-k8s-service",
        "kubernetes_name" : "some-events",
        "kubernetes_namespace" : "deploytest",
        "phase" : "connect",
        "t" : "type",
        "v" : "1"
      }
    }
  }

In query on changing must to should, I get:

"aggregations" : {
  "sum_is" : {
    "value" : 1.5389155527088604E16
  }
}

The index dynamic mapping looks something like this:

"mappings" : {
      "dynamic_templates" : [
        {
          "strings" : {
            "unmatch" : "*seconds*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },
        {
          "to_float" : {
            "match" : "*seconds*",
            "mapping" : {
              "type" : "float"
            }
          }
        }
      ],

However in our requirement, we need results matching all of the clauses in the query.

For metrics aggregation elasticsearch converts everything to double, still this doesn't explain result as zero.

Any pointers will be helpful. Thanks for attention.

NOTE: I see that in example document, value field is zero. Maybe while drafting/editing I made a mistake.

Below is the result of past 2 mins. This shows value field is actually float.

Query:

GET some_metric/_search?size=3
{
"_source": ["value"], 
"query": {
  "bool": {
    "must": [
      {
        "range": { "time": { "gte" : "now-2m", "lt": "now" } }
      },
      {
        "match": {"name": "probe_http_duration_seconds"}
      },
      {
        "match": {"labels.instance": "some-instance"}
      }
    ]
  }
}
}

Result:

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10,
      "relation" : "eq"
    },
    "max_score" : 14.551308,
    "hits" : [
      {
        "_index" : "some_metric-2019.12.04-000005",
        "_type" : "_doc",
        "_id" : "7oog0G4Bk6EPplW1ibD1",
        "_score" : 14.551308,
        "_source" : {
          "value" : 0.040022423
        }
      },
      {
        "_index" : "some_metric-2019.12.04-000005",
        "_type" : "_doc",
        "_id" : "74og0G4Bk6EPplW1ibD1",
        "_score" : 14.551308,
        "_source" : {
          "value" : 3.734E-5
        }
      },
      {
        "_index" : "some_metric-2019.12.04-000005",
        "_type" : "_doc",
        "_id" : "A4og0G4Bk6EPplW1ibH1",
        "_score" : 14.551308,
        "_source" : {
          "value" : 0.015694122
        }
      }
    ]
  }
}

When you use must, how many hits do you see in the response? — Val
For last 5 minutes, with "must" I am getting 20 hits and "should" I am getting 10K hits. — user142148
All of them are higher than 0. Most of the values are after 2nd or 3rd decimal places, like 0.003839167 or 0.014749622. Rarely there are values which are at 5th decimal. Would that make a difference? — user142148
I suspect the type of that field is integer(because the first document that was indexed was 0). Can you double check please? — Val

Val Val · Accepted Answer · 2019-12-04T10:08:20

What you see is just what you indexed in the source document. ES will never modify your source document. However, since the type is long as I thought then it will index that float value as a long and not as a float.

This usually happens when the very first document to be indexed has an integer value, such as 0, for instance.

You can either reindex your data with the proper mapping... Or since you have time-based indexes, just modify the dynamic template and tomorrow's index will be created correctly.

Elasticsearch returns 0.0 for metrics sum aggregation

1 Answers