Elasticsearch aggregate on inner_hits

Question

I am trying to do some aggregations on the inner_hits of a nested object (queries), which are filterated based on the query date. This aggregation I am doing in the following block is aggregating on the main document and all objects in "queries", and not just the ones in the inner hits.

GET /networkcollection/branch_routers/_search/
{
  "_source": false,
  "query": {
    "filtered": {
      "query": {
        "match": {
          "mh": 123
        }
      },
      "filter": {
        "nested": {
          "path": "queries",
          "filter": {
            "range": {
              "queries.dateQuery": {
                "gt": "20160101T200000.000Z",
                "lte": "now"
              }
            }
          },
          "inner_hits": {}
        }
      }
    }
  },
  "aggs": {
    "queries": {
      "filter": {
        "nested": {
          "path": "queries",
          "filter": {
            "range": {
              "queries.dateQuery": {
                "gte": "20160101T200000.000Z",
                "lte": "now"
              }
            }
          }
        }
      },
      "aggs": {
        "minDateQuery": {
          "min": {
            "field": "queries.dateQuery"
          }
        }
      }
    }
  }
}

How can I accomplish this aggregation so that it aggregates only the "queries" objects returned in the inner_hits?

Use Filter Aggregations elastic.co/guide/en/elasticsearch/reference/current/… — Rahul
I am unable to get the syntax correctly, receiving an error that stating "[nested] query does not support [aggs]". — blgrnboy
Can you confirm that your above query is the current one you have and which errors out? — Val

Algorini Algorini · Accepted Answer · 2020-05-14T23:07:38

I'm very late on this answer, but it is very much possible to aggregate only on the inner_hits.

My ES version : 6.2.3

I'm providing a detailed response, with index mapping, a few dummy documents and the search_query + response.

The basic idea is to use the "filter" aggregation. You don't need to actually use the "query" part of the search_request at all, unless you're doing some very complex queries(to narrow down the aggregation profile). Most simple queries can easily be specified in the aggregation "filter".

Index setup:

PUT networkcollection
{
  "mappings": { 
    "branch_routers" : {
      "properties" : {
        "mh" : {
          "type" : "text"
        },
        "queries" : {
          "type" : "nested",
          "properties" : {
            "dateQuery" : {
              "type" : "date"
            }
          }
        }
      }
    }
  }
}

PUT networkcollection/branch_routers/1
{
  "mh" : "corona",
  "queries" : [
    {
      "dateQuery" : "2012-04-23"
    },
    {
      "dateQuery" : "2013-04-23"
    },
    {
      "dateQuery" : "2014-04-23"
    },
    {
      "dateQuery" : "2015-04-23"
    },
    {
      "dateQuery" : "2016-04-23"
    },
    {
      "dateQuery" : "2017-04-23"
    },
    {
      "dateQuery" : "2018-04-23"
    },
    {
      "dateQuery" : "2019-04-23"
    },
    {
      "dateQuery" : "2020-04-23"
    }
  ]
}

PUT networkcollection/branch_routers/2
{
  "mh" : "happy",
  "queries" : [
    {
      "dateQuery" : "2009-04-23"
    },
    {
      "dateQuery" : "2008-04-23"
    },
    {
      "dateQuery" : "2007-04-23"
    },
    {
      "dateQuery" : "2015-04-23"
    },
    {
      "dateQuery" : "2016-04-23"
    },
    {
      "dateQuery" : "2017-04-23"
    },
    {
      "dateQuery" : "2018-04-23"
    },
    {
      "dateQuery" : "2019-04-23"
    },
    {
      "dateQuery" : "2020-04-23"
    }
  ]
}

PUT networkcollection/branch_routers/3
{
  "mh" : "happy",
  "queries" : [
    {
      "dateQuery" : "2001-04-23"
    },
    {
      "dateQuery" : "2008-04-23"
    },
    {
      "dateQuery" : "2007-04-23"
    },
    {
      "dateQuery" : "2015-04-23"
    },
    {
      "dateQuery" : "2016-04-23"
    },
    {
      "dateQuery" : "2017-04-23"
    },
    {
      "dateQuery" : "2018-04-23"
    },
    {
      "dateQuery" : "2019-04-23"
    },
    {
      "dateQuery" : "2020-04-23"
    }
  ]
}

We added three basic documents, now we try to filter on the "mh" as "happy", and we want the minimum dateQuery in the nested objects, such that it filters between the year 2016 and now (We're currently in the middle of the corona-virus lockdown, so you know the year :) ).

Search Query:

GET networkcollection/branch_routers/_search
{
  "_source": false, 
  "query": {
    "match": {
      "mh": "happy"
    }
  },
  "aggs": {
    "filtered_agg": {
      "filter": {
        "match" : {
          "mh" : "happy"
        }
      },
      "aggs": {
        "filtered_nested": {
          "nested": {
            "path": "queries"
          },
          "aggs": {
            "dateQuery_agg": {
              "date_range": {
                "field": "queries.dateQuery",
                "ranges": [
                  {
                    "from": "now-4y/y",
                    "to": "now"
                  }
                ]
              },
              "aggs": {
                "min_date": {
                  "min": {
                    "field": "queries.dateQuery"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

Response:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "networkcollection",
        "_type": "branch_routers",
        "_id": "2",
        "_score": 0.2876821
      },
      {
        "_index": "networkcollection",
        "_type": "branch_routers",
        "_id": "3",
        "_score": 0.2876821
      }
    ]
  },
  "aggregations": {
    "filtered_agg": {
      "doc_count": 2,
      "filtered_nested": {
        "doc_count": 18,
        "dateQuery_agg": {
          "buckets": [
            {
              "key": "2016-01-01T00:00:00.000Z-2020-05-14T23:02:31.611Z",
              "from": 1451606400000,
              "from_as_string": "2016-01-01T00:00:00.000Z",
              "to": 1589497351611,
              "to_as_string": "2020-05-14T23:02:31.611Z",
              "doc_count": 10,
              "min_date": {
                "value": 1461369600000,
                "value_as_string": "2016-04-23T00:00:00.000Z"
              }
            }
          ]
        }
      }
    }
  }
}

As you can see, it correctly filters out the documents listed with "mh" = "corona", and keeps only the two documents with "mh" = "happy", and then filters only those "queries" objects which lie in my specified date range, and finally provides the min_date.

Elasticsearch aggregate on inner_hits

1 Answers