Logstash-plugin elasticsearch: apply executing searches in logstash

Question

Here is my excuting search to query my elasticsearch database (it works fine):

curl -XPOST 'localhost:9200/test/_search?pretty' -d '
{
"size":1,
"query": {
"match": {
"log.device":"xxxx"
}
},
"sort" : [ {
"_timestamp" :
{
"order":"desc"
}
}]
}'

I want to do the same thing through logstash with the plugin elasticsearch. However, there is no "size" option available in the website https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

elasticsearch {
hosts => ["localhost:9200/test"]
query => "log.device:%{[log][device]}"
sort => "@timestamp:desc"
}

Do you how to manage this problem ?

Thank you for your attention and for your help.

Joe

Val Val · Accepted Answer · 2016-06-14T12:12:29

Since the size is hardcoded to 1 in that plugin, you don't have to add any size parameter.

Also make sure to sort on _timestamp not @timestamp.

Finally, the hosts parameter doesn't take any index.

So:

elasticsearch {
   hosts => ["localhost:9200"]
   query => "log.device:%{[log][device]}"
   sort => "_timestamp:desc"
}

If you really need to specify an index, this is not supported yet, but I've created a PR last week in order to support this. So until this gets merged and released, you'll be able to use my version instead of the official one:

$> git clone http://github.com/consulthys/logstash-filter-elasticsearch
$> cd logstash-filter-elasticsearch 
$> gem build logstash-filter-elasticsearch.gemspec
$> $LS_HOME/bin/plugin -install logstash-filter-elasticsearch-2.0.4.gem

After installing the amended plugin, you'll be able to work on a specific index:

elasticsearch {
   hosts => ["localhost:9200"]
   index => "test"
   query => "log.device:%{[log][device]}"
   sort => "_timestamp:desc"
}

Logstash-plugin elasticsearch: apply executing searches in logstash

1 Answers