How to escape keys[ ] while filtering JSON fields using Logstash?

Question

I'm trying to configure Logstash to write some JSON fields into a database.

Source JSON is as follows:

"latencies":{"request":0,"kong":0,"proxy":-1},"service":{"host":"127.0.0.1","created_at":1554311520,"connect_timeout":60000,"id":"bdc5e673-4631-4460-8932-16633057d19f","protocol":"http","name":"Develop","read_timeout":60000,"port":9999,"updated_at":1554316363,"retries":5,"write_timeout":60000},"request":{"querystring":{},"size":"375","uri":"/some/api/32165498/endpoint","url":"http://127.0.0.1:8000/some/api/32165498/endpoint","headers":{"header1":"1","header2":"352515622568","header3":"a","accept":"/","header4":"111","header5":"39318053","cache-control":"no-cache","connection":"keep-alive","accept-encoding":"gzip, deflate","user-agent":"PostmanRuntime/7.13.0","host":"127.0.0.1:8000","postman-token":"e6bdbc21-e39b-4420-9d6f-5666d88d8383","header6":"a","header7":"96"},"method":"GET"},"header7":"127.0.0.1","tries":{},"upstream_uri":"/some/api/32165498/endpoint","response":{"headers":{"connection":"close","content-type":"application/json; charset=utf-8","content-length":"194","server":"kong/1.0.3"},"status":400,"size":"371"},"route":{"created_at":1554311575,"methods":["GET","POST"],"id":"6a13g329-bf1f-451b-bcaa-14cfcf087925","service":{"id":"bd51e673-4631-4460-8932-16633057d19f"},"name":"Develop","hosts":[],"updated_at":1554316427,"preserve_host":false,"regex_priority":0,"paths":[],"protocols":["http"],"strip_path":false},"started_at":1559094275277}

When trying to filter some fields, keys [ ] seem to be messing the output.

Current pipeline's logstash.conf is as follows:

input {
    beats {
    port => "5044"
    host => "0.0.0.0"
    }
}

filter {
    json {
            source => "json-message"
            }

            if [request][header1] {
            mutate {
            add_field => { "header" => "%{request[header1]}"}}}


    mutate {
            remove_field => [ "header" ] 
            }
        }
     }
output {
    file {
    path => "/kong_logs/test.log"
}
}

I have also tried the following configurations to no avail:

add_field => { "header" => "%{[request][header1]}"}}}
add_field => { "header" => "%{[request][0][header1]}"}}}
add_field => { "header" => "%{request[0][header1]}"}}}

With this code I would expect the "header1" field to be filtered out, however it is still in the output.

Thanks in advance!

Andy Burgos Andy Burgos · Accepted Answer · 2019-06-13T19:33:48

Not exactly a solution, but as a workaround I used the following code:

mutate {
            remove_field => [ {"[header1][field]" ]  
}

Worked like a charm.

How to escape keys[ ] while filtering JSON fields using Logstash?

1 Answers