Grok pattern for Custom log line

Question

I am new to grok patterns and I am trying to write grok pattern for below custom log line. I want to extract the values of fields given in log line e.g. ServiceName, SystemDate,SequenceName etc, together with TID,[0] [timestamp]as well. Any help would be great appreciation.

Log:

TID: [0] [ESB] [2016-08-16 10:35:10,828] [jms-Worker-2]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  ServiceName = CustomerService_v1,SystemDate = 8/16/16 10:35 AM,ServerIP = 10.200.42.158,ServerHost = slllasp102.local,SequenceName = SendCustomerToTopic,Message = Going to Send Message to Customer Topic,MessageCode = null,ErrorMessage = null,ErrorDetail = null,ErrorException = null {org.apache.synapse.mediators.builtin.LogMediator}

MyPattern:

\[%{TIMESTAMP_ISO8601:timestamp}\]\s+%{WORD:loglevel}\s+-\s+%{GREEDYDATA:ServiceName}

I am unable to write the correct pattern that retrieve fields one by one.

Please update your question to show what you have already tried. Thank you. — Boris Schegolev
We are not here to do your job, you have to show a problem to solve. cf stackoverflow.com/help/how-to-ask — baudsp
To help you : grokconstructor.appspot.com/do/match to test your grok pattern, github.com/logstash-plugins/logstash-patterns-core/tree/master/… for the patterns, with the basic one in grok-patterns. Also it might be better to use the kv filter to extract the fields (you'll have to first extract that part of the log with grok) — baudsp
stackoverflow.com/users/6113627/baudsp I am not asking to do my job. But community should understand that if some one is beginner then help him. thanks — Waqas Ali Razzaq
@Waqas The response of baudsp was before your edit. Now that you've edited, wait till it gets an answer! (That's why you gotta put all te details at the first itself). Best of luck. — Bhargav Rao♦

baudsp baudsp · Accepted Answer · 2016-09-06T09:29:32

I completed your grok pattern, it should look like this:

TID: \[%{INT:TID}\] \[ESB\] \[%{TIMESTAMP_ISO8601:timestamp}\]\[jms-Worker-2\]\s+%{WORD:loglevel} {%{GREEDYDATA}} - %{GREEDYDATA:fields} {%{GREEDYDATA}}

Then you use the kv filter to extract the fields. It is easier to do it that way than by using the grok filter. The configuration should look like this:

kv {
  source => "field" # That's the field I created in the grok filter containing the fields (ServiceName = CustomerService_v1,SystemDate = 8/16/16 10:35 AM...)
  value_split => "="
  field_split => ","
  trimkey => " "
  trim => " "
}

If you prefer to not to use the kv filter, you'll have to replace the %{GREEDYDATA:fields} by: \s+ServiceName =%{GREEDYDATA:ServiceName},SystemDate =%{GREEDYDATA:SystemDate},...

Grok pattern for Custom log line

1 Answers