logstash multiline codec does not work

Question

I have configured logstash pipeline to report to elastic. I am able to read the log files. My log files contain multiline messages, but each line is being reported as one message to elastic.Following is my logstash configuration file

LogConf file:

input {
        file {
            path => ["abc.log" ]
            start_position => "beginning"
            codec => multiline
            {
                pattern => "^%{LOGLEVEL}"
                negate => "false"
                what => "next"
            }
        }
    }
    filter {

    }
    output {
        # only for debug purposes
        stdout {
            codec => rubydebug
        }
        elasticsearch {
            hosts => ["http://abcd:9200"]
            index => "logstash"
        }
    }

Log file:

DEBUG - LogBO={
    message:############ ##############
------------>>!User Info[################################]
------------>>!Debug Info[ ############################# ]
***************isABCEnabled*********************true
}
DEBUG - LogBO={
    message:############ ##############
------------>>!User Info[################################]
------------>>!Debug Info[ ############################# ]
***************isABCEnabled*********************true
}

I am able to see the logs getting reported to Elastic, but as each line of log is a separate message. I want whole log

DEBUG - LogBO={
    message:############ ##############
------------>>!User Info[################################]
------------>>!Debug Info[ ############################# ]
***************isABCEnabled*********************true
}

to be reported as a single message to Elastic.Please help me fixing the issue.

Please help me fixing this issue.

Will Barnwell Will Barnwell · Accepted Answer · 2017-09-11T23:19:35

By using

codec => multiline {
    pattern => "^%{LOGLEVEL}"
    negate => "false"
    what => "next"
}

You are telling the codec to join any line matching ^%{LOGLEVEL} to join with the next line. This will join the first line to the second line because the first line matches ^%{LOGLEVEL}. The other lines will be ignored and the pattern will not continue matching and joining the same line down. Thus you'll end up with a mess of partial log events.

You will want to use

codec => multiline {
    pattern => "^%{LOGLEVEL}"
    negate => "true"
    what => "previous"
}

instead. This tells logstash to join any line that does not match ^%{LOGLEVEL} to the previous line. This ensures that events always start with a ^%{LOGLEVEL} matching line and is what you want.

logstash multiline codec does not work

1 Answers