How to create field using Logstash and grok plugin

Question

I have a tomcat log of below format

10.0.6.35 - - [21/Oct/2019:00:00:04 +0000] "GET /rest/V1/productlist/category/4259/ar/final_price/asc/4/20 HTTP/1.1" 200 14970 12

I want to create the field of last two column which is bytes and duration and want to analyze it using Kibana. I had used Filebeat and Logstash for transferring data to the Elasticsearch.

My Logstash configuration file is below:

I had tried with below configuration but not able to see the field on kibana.

input {
     beats {
     port => 5044
  }
 }

filter {
  grok {
  match => ["message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes}(?m) %{NUMBER:duration}" ]
#match=>{"duration"=> "%{NUMBER:duration}"}
# match => { "message" => "%{COMBINEDAPACHELOG}" }

  }
#  mutate {
#    remove_field => ["@version", "@timestamp"]
#  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
if [fields][log_type] == "access-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
if [fields][log_type] == "application-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
else
{
  elasticsearch {
    hosts => ["172.31.30.73:9200"]
    index => "logstashhh-%{+YYYY.MM.dd}"
}

I want that duration and bytes becomes my field on Kibana for visualization.

Do you have any example of a document inputed to Kibana with this .conf of yours? How it looks like? May you edit your question, showing us a document, in JSON? — Lodi
Please read Under what circumstances may I add “urgent” or other similar phrases to my question, in order to obtain faster answers? - the summary is that this is not an ideal way to address volunteers, and is probably counterproductive to obtaining answers. Please refrain from adding this to your questions. — halfer
The provided pattern didn't match the example log line. You can use instead %{NUMBER:bytes}(?m) %{NUMBER:duration}$ — baudsp
Also perhaps reload your index field list in Kibana (elastic.co/guide/en/kibana/7.4/…). — baudsp

Ishara Madhawa Ishara Madhawa · Accepted Answer · 2019-10-23T14:32:04

Try this as your logstash configuration:

input {
     beats {
     port => 5044
  }
 }

filter {
  grok {
  match => ["message" => "%{NUMBER:bytes}(?m) %{NUMBER:duration}$" ]
#match=>{"duration"=> "%{NUMBER:duration}"}
# match => { "message" => "%{COMBINEDAPACHELOG}" }

  }
#  mutate {
#    remove_field => ["@version", "@timestamp"]
#  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
if [fields][log_type] == "access-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
if [fields][log_type] == "application-log"
{
elasticsearch {
  hosts => ["172.31.30.73:9200"]
  index => "%{[fields][service]}-%{+YYYY.MM.dd}"
 }
}
else
{
  elasticsearch {
    hosts => ["172.31.30.73:9200"]
    index => "logstashhh-%{+YYYY.MM.dd}"
}

How to create field using Logstash and grok plugin

1 Answers