Filebeat not sending correct multiline log to logstash

1
votes

For some reason filebeat is not sending the correct logs while using the multiline filter in the filebeat.yml file. The log file im reading has some multiline logs, and some single lines. However, they all follow the same format by starting with a date. For an example, here is a couple lines:

2017-Aug-23 10:33:43: OutputFile: This is a sample message 
2017-Aug-23 10:34:23: MainClass: Starting connection:
   http.InputProcess: 0
   http.OutPutProcess: 1
2017-Aug-23 10:35:21: OutputFile: This is a sample message 2

My Filebeat yml is:

- input_type: log
  paths:
  - /home/user/logfile.log
  document_type: chatapp
  multiline:
    pattern: "^%{YYYY-MMM-dd HH:mm:ss}"
    negate: true
    match: before

For some reason when i see the filebeat logs hit elasticsearch, all of the logs will be aggragated into one log line, so it does not seem to be actually reading the file date by date. Can Anyone help? Thanks!

1
logstashfilebeat

1 Answers

2
votes

Use 

pattern: "^%{YEAR}-%{MONTH}-%{MONTHDAY}"

The pattern you are currently using there is not a validly defined regex given the grok patterns.

You can test multiline patterns using the grokconstructor. I constructed this pattern from the grok-patterns predefined in logstash.