Grok Pattern - for comma delimited data

Question

I'm trying to capture a Grok pattern for the following data record:

-- One row --

0,2018-12-17 22:40:30.000,25980000,92,True,0,400,33,0,2018-12-16 22:40:30.000,2018-12-17 05:53:30.000,433,17,32,1,1,18,2018-12-17,Very Awake

-- Second Row --

1,2018-12-17 22:41:30.000,25980000,92,True,0,400,33,0,2018-12-16 22:40:30.000,2018-12-17 05:53:30.000,433,17,32,1,1,18,2018-12-17,Awake

Here is my grok pattern:

%{NUMBER:ID}%{NOTSPACE}%{TIMESTAMP_ISO8601:RecordedDateTimeStamp}*%{NOTSPACE}%{NUMBER:Efficiency}%{NOTSPACE}%{DATA:IsMainSleep}%{NOTSPACE}%{NUMBER:MinutesAfterWakeup}%{NOTSPACE}%{NUMBER:MinutesAsleep}%{NOTSPACE}%{NUMBER:MinutesAwake}%{NOTSPACE}%{NUMBER:MinutesToFallAsleep}%{NOTSPACE}%{NUMBER:SleepStartTime}%{NOTSPACE}%{NUMBER:SleepEndTime}%{NOTSPACE}%{NUMBER:TimeInBed"}%{NOTSPACE}%{NUMBER:RestlessCount}%{NOTSPACE}%{NUMBER:RestlessDuration}%{NOTSPACE}%{NUMBER:AwakeCount}%{NOTSPACE}%{NUMBER:AwakeDuration}%{NOTSPACE}%{NUMBER:AwakeningsCount}%{NOTSPACE}%{TIMESTAMP_ISO8601:DateOfSleep}%{NOTSPACE}%{DATA:SleepState}

I'm not sure why, but it is not working in the Grok Debugger tool in Kibana.

Can anyone help me figure out what is wrong with my grok pattern?

Thank you in advance

leandrojmp leandrojmp · Accepted Answer · 2018-12-27T19:44:59

Your grok pattern has 18 fields, but your data records have 19 fields, the third field, which has the same value on both records (25980000, is this a job id?), wasn't being considered in your pattern.

Also, you have two others timestamps that weren't being parsed as timestamps and you need to anchor the last field using $ to tell where each message ends (where grok will strop trying to match anything)

Try this grok pattern:

%{NUMBER:ID},%{TIMESTAMP_ISO8601:RecordedDateTimeStamp},%{NUMBER:jobId},%{NUMBER:Efficiency},%{DATA:IsMainSleep},%{NUMBER:MinutesAfterWakeup},%{NUMBER:MinutesAsleep},%{NUMBER:MinutesAwake},%{NUMBER:MinutesToFallAsleep},%{TIMESTAMP_ISO8601:SleepStartTime},%{TIMESTAMP_ISO8601:SleepEndTime},%{NUMBER:TimeInBed},%{NUMBER:RestlessCount},%{NUMBER:RestlessDuration},%{NUMBER:AwakeCount},%{NUMBER:AwakeDuration},%{NUMBER:AwakeningsCount},%{DATA:DateOfSleep},%{DATA:SleepState}$

Just tested here on my lab and got this result:

{
"MinutesToFallAsleep": "0",
"MinutesAsleep": "400",
"AwakeningsCount": "18",
"TimeInBed": "433",
"AwakeDuration": "1",
"SleepEndTime": "2018-12-17 05:53:30.000",
"MinutesAfterWakeup": "0",
"RestlessDuration": "32",
"jobId": "25980000",
"MinutesAwake": "33",
"SleepStartTime": "2018-12-16 22:40:30.000",
"Efficiency": "92",
"AwakeCount": "1",
"IsMainSleep": "True",
"RestlessCount": "17",
"RecordedDateTimeStamp": "2018-12-17 22:41:30.000",
"DateOfSleep": "2018-12-17",
"ID": "1",
"SleepState": "Awake"
}

Grok Pattern - for comma delimited data

1 Answers