.net logs from filebeat to logstash multiline proper format

Question

I'm trying to ingest logs from a .net application. I have filebeat installed on a node which pushes the logs to a logstash server.

logfile:

2020-06-19 00:00:16.421 +02:00 [Error] [Band.Account.HealthCheckService] [2HB0AJ9Q9AI2O:00000001] The operation was canceled.
System.Threading.Tasks.TaskCanceledException: The operation was canceled. ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled. ---> System.Net.Sockets.SocketException: Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
   at System.Net.Security.SslStreamInternal
...

filebeat.yml config:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/log*
  multiline.pattern: '^[[:space:]]'
  multiline.negate: false
  multiline.match: after
  fields_under_root: true
  fields:
    type: Band-account-log
  fields_under_root: true
  exclude_files: ['.gz$']
logging.level: info
output.logstash:
  hosts: ["elk.Band.net"]
  ssl.certificate_authorities: "/etc/pki/tls/certs/logstash-forwarder.crt"

It doesn't really get all the lines from log file, and the timestamp looks wrong looking in kibana. Not really sure what I need to modify.

filebeat version 6.2.4 (amd64), libbeat 6.2.4*

catalin catalin · Accepted Answer · 2020-06-19T13:09:50

I managed to make it work using the conf below:

filebeat.yml:

filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /var/log/log*
  multiline.pattern: '([12]\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01]))'
  multiline.negate: true
  multiline.match: after
  fields_under_root: true
  fields:
    type: pirelli-account-log
  fields_under_root: true
  exclude_files: ['.gz$']
logging.level: info
output.logstash:
  hosts: ["elk.server.example"]
  ssl.certificate_authorities: "/etc/pki/tls/certs/logstash-forwarder.crt"

Now the logs are coming in the proper format.

.net logs from filebeat to logstash multiline proper format

1 Answers