Logstash Grok errors

Question

I got this following logs which is json format, what is the best way to grok it, so that I have field created for Key. Thank you for your time.

Logs:

2018-10-17 16:20:04,358 WARNING VID_DROPS {"JITTER": 0.1, "INTVL": 6, "DATE": "Wed Oct 17 15:53:45 2018", "SOURCEIP": "192.168.12.1:22100", "ERRORS": 0.02, "LOSTPKT": 34, "FLOW": 116288, "MCAST": "239.0.1.102:1000", "SWITCH": "switc01", "INTERFACE": "TenGigE0/0/2/0", "CLASS": "Policy_VID"}

Here is the filter I got, seems not working:

     grok  {
   match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:loglevel} %{WORD:VID_DROPS} %{NOTSPACE:json1}" }
   remove_field => [ "message" ]
  } 
json { source => "json1" remove_field => [ "json1" ] }

Use GREEDYDATA instead of NOTSPACE for json1. To test your patterns: grokconstructor.appspot.com/do/match. All the basic patterns are there: github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns — baudsp

Sufiyan Ghori Sufiyan Ghori · Accepted Answer · 2018-10-18T21:21:27

AS baudsp mentioned, You need to use GREEDYDATA in order to match everything after the word VID_DROPS. Beside, there is a default pattern available to match loglevel, %{LOGLEVEL:loglevel} so you don't need to use WORD,

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{WORD:VID_DROPS} %{GREEDYDATA:json1}

Will output,

{
  "timestamp": [
    [
      "2018-10-17 16:20:04,358"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "10"
    ]
  ],
  "MONTHDAY": [
    [
      "17"
    ]
  ],
  "HOUR": [
    [
      "16",
      null
    ]
  ],
  "MINUTE": [
    [
      "20",
      null
    ]
  ],
  "SECOND": [
    [
      "04,358"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "loglevel": [
    [
      "WARNING"
    ]
  ],
  "VID_DROPS": [
    [
      "VID_DROPS"
    ]
  ],
  "json1": [
    [
      "{"JITTER": 0.1, "INTVL": 6, "DATE": "Wed Oct 17 15:53:45 2018", "SOURCEIP": "192.168.12.1:22100", "ERRORS": 0.02, "LOSTPKT": 34, "FLOW": 116288, "MCAST": "239.0.1.102:1000", "SWITCH": "switc01", "INTERFACE": "TenGigE0/0/2/0", "CLASS": "Policy_VID"}"
    ]
  ]
}

You can test it at https://grokdebug.herokuapp.com/

Logstash Grok errors

1 Answers