Elasticsearch query fails in logstsah: was expecting a colon to separate field name and value

Question

I've been trying to use logstash to query for elasticsearch events so that I could fill up some fields with more human readable data. I have one index for person names, which I try to use to enhance other fields when I'm indexing other kinds of events.

I quess there is something wrong with my query string, but I don't know what. I just keep getting this warning and the person_query_failed tag:

[2019-10-03T10:38:52,759][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-people", :error=>"Unexpected character ('}' (code 125)): was expecting a colon to separate field name and value\n at [Source: (byte[])\"{\n \"query\": {\n \"query_string\": { \"query\": { \"id:109\" } }\n },\n \"_source\": { \"cn\" }\n}\"; line: 3, column: 42]"}

[2019-10-03T10:38:52,760][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-people", :error=>"Unexpected character ('}' (code 125)): was expecting a colon to separate field name and value\n at [Source: (byte[])\"{\n \"query\": {\n \"query_string\": { \"query\": { \"id:4\" } }\n },\n \"_source\": { \"cn\" }\n}\"; line: 3, column: 40]"}

[2019-10-03T10:38:52,764][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-people", :error=>"Unexpected character ('}' (code 125)): was expecting a colon to separate field name and value\n at [Source: (byte[])\"{\n \"query\": {\n \"query_string\": { \"query\": { \"id:49\" } }\n },\n \"_source\": { \"cn\" }\n}\"; line: 3, column: 41]"}

This is my query template person_query.json:

{ 
  "query": { 
    "query_string": { "query": { "id:%{[dn_people]}" } }
  },
  "_source": { "cn" } 
}

And here is my logstash filter configuration:

filter {
    elasticsearch {
        hosts => [ "https://localhost:9200" ]
        index => "logstash-people"
        query_template => "/path/to/person_query.json"
        fields => { "cn" => "person" }
        user => admin
        password => password
        ca_file => "/etc/elasticsearch/root-ca.pem"
        tag_on_failure => person_query_failed
    }
}

When looking at the warn logs, at least the query string itself seems to function as it should: "id:%{[dn_people]}" translates to \"id:109\", \"id:4\" and \"id:125\" respectively.

When writing my query, I've been looking at this: https://discuss.elastic.co/t/how-to-query-elasticsearch-and-take-some-fields-from-old-data/75300/3

Could someone help and push me in the right direction in fixing this?

I'm using Opendistro for Elasticsearch.

Answer 1

In my opinion the problem lies here

Unexpected character ('}' (code 125)): was expecting a colon to separate field name and value\n at [Source: (byte[])\"{\n \"query\": {\n \"query_string\": { \"query\": { \"id:49\" } }\n },\n \"_source\": { \"cn\" }\n}\"; line: 3, column: 41]"}

Your query template should look like this:

{
  "query": { 
    "query_string": { "query": { "id": %{[dn_people]} } }
  },
  "_source": { "cn" } 
}

The difference is that you need to seperate the field and the value by quoting the entire field name followed by a colon and then the value.

You can identify the problem by looking at the resolved query:

\"id:109\"

Without the escape characters it looks like this:

"id:109"

Elasticsearch interprets the whole string as the field name and can not find the following colon and value.

It has to be resolved to:

\"id\":109 or unescaped: "id":109

Answer 2

Thanks to apt-get_install_skill's answer I was able to move forward with my problem. There were several, which I could easily resolve with some heavy googling.

Lastly, when using this query template

 { 
   "query": {
     "query_string": { "query": { "id": %{[dn_people]} } } 
   },
   "_source": [ "cn" ] 
}

this warning appeared:

[2019-10-03T12:50:20,563][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"logstash-people", :error=>"[400] {\"error\":{\"root_cause\":[{\"type\":\"parsing_exception\",\"reason\":\"[query_string] unknown token [START_OBJECT] after [query]\",\"line\":1,\"col\":35}],\"type\":\"parsing_exception\",\"reason\":\"[query_string] unknown token [START_OBJECT] after [query]\",\"line\":1,\"col\":35},\"status\":400}"}

I found this thread Unknown token error for elastic search query which did not give a straight answer to my situation, but fortunately baudsp had already given the right answer as a comment.

So here is the proper query template as per comment from baudsp:

{ 
  "query": {
    "query_string": { "query": "id:%{[dn_people]}" }
  },
  "_source": [ "cn" ]
}