Insert aggregation results into an index

Question

The goal is to build an Elasticsearch index with only the most recent documents in groups of related documents to track the current state of some monitoring counters and states.

I have crafted a simple Elasticsearch aggregation query:

{
  "size": 0,
  "aggs": {
    "group_by_monitor": {
      "terms": {
        "field": "monitor_name"
      },
      "aggs": {
        "get_latest": {
          "top_hits": {
            "size": 1,
            "sort": [
              {
                "timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        }
      }
    }
  }
}

It groups related documents into buckets and select the most recent document for each bucket.

Here are the different ideas I had to get the job done:

directly use the aggregation query to push the results into the index, but it does not seem possible : Is it possible to put the results of an ElasticSearch aggregation back into the index?
use the Logstash Elasticsearch input plugin to execute the aggregation query and the Elasticsearch output plugin to push into the index, but seems like the input plugin only looks at the hits field and is unable to handle aggregation results: Aggregation Query possible input ES plugin !
use the Logstash http_poller plugin to get a JSON document, but it does not seem to allow specifying a body for the HTTP request !
use the Logstash exec plugin to execute cURL commands to get the JSON but this seems quite cumbersome and my last resort.
use the NEST API to build a basic application that will do polling, extract results, clean them and inject the resulting documents into the target index, but I'd like to avoid adding a new tool to maintain.

Is there a reasonably complex way of accomplishing this?

@AndreiStefan Thanks but AFAIK Watcher won't help for this use case. Moreover we don't have it (yet?) deployed on our infrastructure. For alerting we use ElastAlert which does the job perfectly too. — Pragmateek
I'm not suggesting Watcher for alerting, but for being able to query the indices at a regular interval, do some basic transformation on the resulted data and be able to index back into Elasticsearch. — Andrei Stefan
@AndreiStefan Thanks for these elements. Indeed Watcher seems a good alternative. But as said before we don't have it yet. :'( — Pragmateek
@OvidiuRudi Not a direct way, I had to build a dedicated program in C# to make the plumbing. — Pragmateek

Akshay Patil Akshay Patil · Accepted Answer · 2017-02-06T10:45:22

Edit the logstash.conf file as follow

input {
  elasticsearch {
    hosts => "localhost" 
    index => "source_index_name" 
    type =>"index_type" 
    query => '{Query}' 
    size => 500 
    scroll => "5m" 
    docinfo => true
  }
}

output { 
  elasticsearch { 
    index => "target_index_name" 
    document_id => "%{[@metadata][_id]}"
  }
}

Insert aggregation results into an index

1 Answers