How to split Multiline log in logstash

Question

I am trying to create filter for my Log file. My log is:

    =-=-=-=-=-=-=-=-
    Timestamp: Thursday, April 19, 2018 2:48:49 AM
    Message: HandlingID: 3
    An exception of type 'System.Exception' occurred and was caught.
    ----------------------------------------------------------------
    04/19/2018 02:48:49
    Type : System.Exception,ib, Version=4.0.0.0, Culture=neutral
    Message : TRY
    Source : 
    Help link : 
    Data : LinkedList
    TargetSite : 
    HResult : LALA
    Stack Trace : The stack trace is unavailable.
    Additional Info:

    MachineName : S
    TimeStamp : 4/19/2018 6:48:49 AM
    FullName : Some Info
    AppDomainName : AA
    ThreadIdentity : 
    WindowsIdentity : jj

    Category: Error
    Priority: 0
    EventId: 1
    Severity: Extreme
    Title:p
    Machine: S
    Application Domain: y
    Process Id: 
    Process Name: l
    Win32 Thread Id: 6
    Thread Name: 
    Extended Properties: 
    =-=-=-=-=-=-=-=-
    =-=-=-=-=-=-=-=-

//SIMILAR LOG
    =-=-=-=-=-=-=-=-

Here =-=-=-=-=-=-=-=- denotes the end and starting of a new log. My config file is:

input {
beats {
        port => "5044"
    }
}
filter {
multiline {
       pattern => "^=-=-=-=-=-=-=-=-"
       negate => true
       what => previous
    }
}
output {
elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

This is creating 2 documents(rows) for each log. One containing separator+log and other contains separator only. I want to delete the document containing only separator. Also , Please let me know how to split my log into different fields and if there is any proper documentation which could help me forming filters . I am new to logstash.

Sufiyan Ghori Sufiyan Ghori · Accepted Answer · 2018-06-13T10:34:45

If you want to include all the new lines and split your log on each occurrence of =-=-=-=-=-=-=-=-, you can match it with multi-line modifier, (?m) as follows,

(?m)%{GREEDYDATA:log}=-=-=-=-=-=-=-=-

It will output,

{
  "log": [
    [
      "    Timestamp: Thursday, April 19, 2018 2:48:49 AM\n    Message: HandlingID: 3\n    An exception of type 'System.Exception' occurred and was caught.\n    ----------------------------------------------------------------\n    04/19/2018 02:48:49\n    Type : System.Exception,ib, Version=4.0.0.0, Culture=neutral\n    Message : TRY\n    Source : \n    Help link : \n    Data : LinkedList\n    TargetSite : \n    HResult : LALA\n    Stack Trace : The stack trace is unavailable.\n    Additional Info:\n\n    MachineName : S\n    TimeStamp : 4/19/2018 6:48:49 AM\n    FullName : Some Info\n    AppDomainName : AA\n    ThreadIdentity : \n    WindowsIdentity : jj\n\n    Category: Error\n    Priority: 0\n    EventId: 1\n    Severity: Extreme\n    Title:p\n    Machine: S\n    Application Domain: y\n    Process Id: \n    Process Name: l\n    Win32 Thread Id: 6\n    Thread Name: \n    Extended Properties: \n    "
    ]
  ]
}

You can test it at, https://grokdebug.herokuapp.com/

How to split Multiline log in logstash

1 Answers