Extract from LogStash message field in Kibana dashboard

Question

I have configured a LogStash EC2 server to push some logs into an AWS ElasticSearch domain. The process is working alright and the logs are visible in Kibana. However, I need to extract data inside the field called 'message' and show those extracted data in separate columns in the Kibana dashboard. I've followed some tutorials and tried modifying the logstash configuration with grok tool.

Each record has about 35 fields, and the field 'message' is a complex field which consists of 10 separate values. What I need is to extract each of those 10 values for each record, and present them as separate fields.
The following is an example field for the message.

2021-04-05 13:32:03.746+0000 | INFO | BRAINWAVE_API | DEV |  |  | 22460 | [scheduling-1] | c.h.w.a.bwService | Fetching results for orgI : 81ge0-de12ff-59jla0 orgN : Cactus Corp

Below is my current grok configuration inside the logstash.conf file. Note that I am trying to extract the value of the first sub-field in message field first, therefore only one parameter :

filter {
  grok {
   match => { "message" => "%{NUMBER:yearField}" }
 }
}

It returns an extra field named 'yearField', but it only contains '2021' which I assume it's only able to catch the first part of the first sub-field in the 'message' field 2021-04-05 13:32:03.746+0000.

What am I doing wrong with this grok configuration?

Sourav Atta Sourav Atta · Accepted Answer · 2021-04-07T07:59:58

Here is the grok pattern that will match the log pattern:

%{TIMESTAMP_ISO8601:timestamp} \| %{LOGLEVEL:loglevel} \| %{WORD:Api} \| %{WORD:user} \| %{SPACE:} \| %{SPACE} \| %{POSINT:bytes} \| \[%{DATA:thread}\] \| %{DATA:service} \| %{GREEDYDATA:message}

I have used the Grok Debugger to test the pattern.

Extract from LogStash message field in Kibana dashboard

1 Answers