Filebeat - Setting up a multiline configuration

Question

I have one filebeat that reads severals different log formats.

One format that works just fine is a single liner, which is sent to Logstash as a single event. Now, I have another format that is a multiliner. I want to read it as a single event and send it to Logstash for parsing. This is the log format example, with two events.

error: I READ THIS. sent payload: [{"key": "values"}]
custom status response: [{"key1": "values"}]
callback headers:  [{"key2": "values"}]
error stack: [ something really bad happened
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)]

error: I AM NOT READING THIS. sent payload: [{"key": "values"}]
custom status response: [{"key1": "values"}]
callback headers:  [{"key2": "values"}]
error stack: [ something really bad happened
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)
    at here loremisptul (/xx/xx/x)]

And here is the prospector configuration:

- input_type: log
  paths: /Users/xxxxx/Downloads/elk/anotherlog/app.stderr.log
  document_type: logsystemtwo
  multiline.pattern: '`^=[A-Z]+|^$`'
  multiline.negate: true
  multiline.match: after
- input_type: log
  paths: /Users/xxxxx/Downloads/elk/mylogs/access.log*
  document_type: logsystemone

The issue is with the first multiline. It reads first event (I READ THIS), but ignores rest of it (I AM NOT READING THIS).

I have tried many different configurations, but I just cant get it to read other events. It always sends only first event, and ignores the rest.

I know I could do the multiline inside Logstash as well, but according to the documentation that should be avoided. Also in my case, I would avoid doing this in Logstash, as I already have very complex structures there

Wexoni Wexoni · Accepted Answer · 2017-05-12T08:24:14

I have found a pattern that works in my case:

- input_type: log
  paths: /Users/xxxxx/Downloads/elk/logs/app.stderr.log
  document_type: error
  multiline.pattern: '^error: '
  multiline.negate: true
  multiline.match: after

Would like to mention there is a playground for the Filebeat, which can help prototyping the patterns.

Filebeat - Setting up a multiline configuration

1 Answers