How to process JSON nested array in Logstash

Question

I have a nested field with arrays in array in JSON like the following:

The following is my config file:

input {
    file {
        codec => "json"
        path => "pathtofile"
        type => "footype"
        start_position => "beginning"
    }
}
filter {
    json {
        source => "message"
        remove_field => [ "host", "message", "path" ]
    }
}
output {
    elasticsearch {
        action => "index"
        index => "bar"
        hosts => [ "http://localhost:9200" ]
    }
}

I got the following error:

09:40:47.725 [[main]>worker0] WARN logstash.outputs.elasticsearch - Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"bar", :_type=>"footype", :_routing=>nil}, 2017-02-13T01:40:30.387Z myconnection %{message}], :response=>{"index"=>{"_index"=>"bar", "_type"=>"footype", "_id"=>"AVo1IN0vK2jgwdCXqZ-q", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [foo.bar] of different type, current_type [long], merged_type [text]"}}}}

I have a feeling that it's the array problem. I have done some research and know that array is not well supported. But I need to ingest the array in elasticsearch. Is there a way to actually do that?

Any helps will be appreciated.

Yi Xu Chee Yi Xu Chee · Accepted Answer · 2017-02-14T06:27:02

I solved this by using a ruby filter:

ruby {
        code => '
            j = 0
            for i in event.get("[foo][bar]") do
                #i is an array element in the big array
                l = 0
                for k in i do
                    event.set("item#" + j.to_s + "#" + l.to_s, k)
                    l = l + 1
                end
                j = j + 1
            end
        '
    }

This will eventually produce fields

item#0#0 = "a"
item#0#1 = "b"
item#1#0 = "c"
item#1#1 = "d"

How to process JSON nested array in Logstash

1 Answers