Does Logstash support Elasticsearch's _update_by_query?

Question

Does the Elasticsearch output plugin support elasticsearch's _update_by_query? https://www.elastic.co/guide/en/logstash/6.5/plugins-outputs-elasticsearch.html https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html

Answer 1

The elasticsearch output plugin can only make calls to the _bulk endpoint, i.e. using the Bulk API.

If you want to call the Update by Query API, you need to use the http output plugin and construct the query inside the event yourself. If you explain what you want to achieve, I can update my answer with some more details.

Note: There's an issue requesting this feature, but it's still open after two years.

UPDATE

So if your input event is {"cname":"wang", "cage":11} and you want to update by query all documents with "cname":"wang" to set "cage":11, your query needs to look like this:

POST your-index/_update_by_query
{
  "script": {
    "source": "ctx._source.cage = params.cage",
    "lang": "painless",
    "params": {
      "cage": 11
    }
  },
  "query": {
    "term": {
      "cname": "wang"
    }
  }
}

So your Logstash config should look like this (your input may vary but I used stdin for testing purposes):

input {
  stdin {
    codec => "json"
  }
}
filter {
  mutate {
    add_field => {
      "[script][lang]" => "painless"
      "[script][source]" => "ctx._source.cage = params.cage"
      "[script][params][cage]" => "%{cage}"
      "[query][term][cname]" => "%{cname}"
    }
    remove_field => ["host", "@version", "@timestamp", "cname", "cage"]
  }
}
output {
  http {
    url => "http://localhost:9200/index/doc/_update_by_query"
    http_method => "post"
    format => "json"
  }
}

Answer 2

The same result can be obtained with standard elasticsearch plugins:

input {
    elasticsearch {
        hosts => "${ES_HOSTS}"
        user => "${ES_USER}"
        password => "${ES_PWD}"
        index => "<your index pattern>"
        size => 500
        scroll => "5m"
        docinfo => true
    }
}

filter {
    ...
}

output {
    elasticsearch {
        hosts => "${ES_HOSTS}"
        user => "${ES_USER}"
        password => "${ES_PWD}"
        action => "update"
        document_id => "%{[@metadata][_id]}"
        index => "%{[@metadata][_index]}"
    }
}