Logstash Beats Input - multiple multiline codec

Question

I am trying to send multiple types of logs with beats and parse them on the logstash server.

I have beats configured and working properly and almost have logstash working correctly.

Where I am having issues is that other-log.log has entries that start with a different format string.

In an ideal world I would like to be able to apply a different multiline codec depending on the type of entry.

I have tried

        if [type] == "server.log" {
            codec => multiline {
                    pattern => "^\d{2}:\d{2}:\d{2},\d+"
                    negate => true
                    what => "previous"
            }
        }

however that causes logstash to fail, my guess is that if is not allowed in the input block.

I have also tried to use the multiline filter plugin but it results in

"Couldn't find any filter plugin named 'multiline'. Are you sure this is correct? Trying to load the multiline filter plugin resulted in this error: LoadError"

Does anyone have an idea as to how to make this work?

filebeat.yml

- input_type: log
  paths:
    - /application/server.log
  document_type: server.log
- input_type: log
  paths:
    - /tmp/other-log.log
  document_type: other.log

pipeline.conf

input {
    beats {
        host => "0.0.0.0"
        port => "5044"
        codec => multiline {
                pattern => "^\d{2}:\d{2}:\d{2},\d+"
                negate => true
                what => "previous"
        }
    }
}
filter {
        if [type] == "server.log" {
                grok {
                        match => { "message" => "(?<date>^\d{2}:\d{2}:\d{2},\d+)\s(?<level>[A-Z]+)\s+\[(?<class>.*?)\]\s+(?<message>(?m).*)" }
                        overwrite => ["message"]
                        add_tag =>  [ "server.log" ]
                }
        }
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
        elasticsearch { hosts => ["localhost:9200"] }
}

xandout xandout · Accepted Answer · 2016-12-08T19:24:08

2

votes

I moved the multiline to filebeat.yml and that solved my issues :)

Logstash Beats Input - multiple multiline codec

1 Answers