Logstash - FileBeat multiline setting groups too much messages

0
votes

I've configured FileBeat to send multiline logs using the following config:

-
  paths:
    - /opt/wls/domains/filename.log
  input_type: log
  document_type: log_doc
  multiline:
    pattern: ^%{TIMESTAMP_ISO8601}
    negate: true
    match: after

As I know, it should append all the lines to the previous one, until it finds a line, which starts with a timestamp (TIMESTAMP_ISO8601).

In my case with this setting filebeat sends several log messages grouped to one single. My log messages start like this:

2016-10-14 20:31:07,447 INFO [ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' ...

It should match to ^%{TIMESTAMP_ISO8601}, so what can be the problem? Why are they sent as one message?

Thank You.

P.S. I've also tried with ^%{YYYY} and ^%{YEAR}patterns, but the result was the same...

2
elasticsearchlogstashmultilinefilebeat
All the examples in the doc have the regex pattern in single quotes. Also, a quick search shows no examples that use the named patterns from logstash in Filebeat. Try a regular regexp. – Alain Collins
Are you sure that filebeat can use the logstash grok pattern? I haven't see anything about it in the doc – baudsp
You'll have to write a regex, perhaps using github.com/logstash-plugins/logstash-patterns-core/blob/master/… – baudsp

2 Answers

1
votes

As written in the comments, FileBeat doesn't support grok patterns. I wrote a regexp instead of the grok pattern, and it worked well. The supported regexps can be found here: https://www.elastic.co/guide/en/beats/filebeat/1.2/regexp-support.html and some multiline examples and tips in case of FileBeat can be found here: https://www.elastic.co/guide/en/beats/filebeat/1.2/multiline-examples.html

0
votes

Obviously GROK will not help and have to use Regex. What I did was as below and it works for me,

filebeat.prospectors:

    - type: log

      enabled: true

      paths:
          - /xxx/server.log*

      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

Here I'm just looking for time stamp at the beginning of the line.