logstash - loading a single-line log and multi-line log at the same time

Question

I'm loading data from two log-files to elasticsearch. First lines of my logstash-file.conf:

input {
    file {
        path => ["/LOGS/BBC/current_log.log",
            "/LOGS/CSI/current_log.log"]
        start_position => "beginning"
}

First ("/LOGS/BBC/current_log.log") is single-line file and loading in elastic whithout errors. Second ("/LOGS/CSI/current_log.log") is multi-line. I want use Multiline codec plugin, but I can't understand - how separate single-line file from multi-line file in input section.

apt-get_install_skill apt-get_install_skill · Accepted Answer · 2020-01-24T09:24:36

You're right that you can only apply one codec-option per input. That being said, nothing stops you from defining multiple input-plugins. So you can specify one input-plugin for your single-line files and another for your multi-line files like so:

logstash.conf:

input {
  file {
    path => ["/LOGS/BBC/current_log.log"]
    start_position => "beginning"
}

input {
  file {
    path => ["/LOGS/CSI/current_log.log"]
    start_position => "beginning"
    codec => "multiline"
}

filter{
  ...
}

output{
  ...
}

Be aware that if one of the input-plugins fail, the whole pipeline gets blocked. So you might want to think about separating your pipelines by defining multiple pipelines. Take a look at this documentation about implementing multiple pipelines.

logstash - loading a single-line log and multi-line log at the same time

1 Answers