Logstash grok filter debugging

Question

Please help, I'm trying to add grok filter in my Logstash pipeline which will convert below logline

2020-11-06 12:57:43,854 INFO Bandwidth: NASDAQ:224.0.130.65:30408 0.000059 Gb/S

to

{
  "ts": [
    [
      "2020-11-06 12:57:43,854"
    ]
  ],
  "YEAR": [
    [
      "2020"
    ]
  ],
  "MONTHNUM": [
    [
      "11"
    ]
  ],
  "MONTHDAY": [
    [
      "06"
    ]
  ],
  "HOUR": [
    [
      "12",
      null
    ]
  ],
  "MINUTE": [
    [
      "57",
      null
    ]
  ],
  "SECOND": [
    [
      "43,854"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "loglevel": [
    [
      "INFO"
    ]
  ],
  "Metric": [
    [
      "Bandwidth"
    ]
  ],
  "Chanel": [
    [
      "NASDAQ:224.0.130.65:30408"
    ]
  ],
  "Data": [
    [
      "0.000059 Gb/S"
    ]
  ]
}

and below is my grok filter

input{
  beats{
    port => "5044"
  }
}

filter{
  if "Bandwidth" in [message]{
    grok{
      match => {"message" => "%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:loglevel} %{WORD:Metric}: (?<Chanel>[A-Z]+:[0-9]+.[0-9]+.[0-9]+.[0-9]+:[0-9]+)"}
    }
  }
}

output{
  elasticsearch{
    hosts => [ "localhost:9200" ]
  }
}

This filter works perfectly fine when I try it in Grok debugger but not in Logstash when viewed in Kibana. I don't see any name captures from filter. Just the message. If I remove the regex part of filter and add GREEDYDATA, everything works. I'm sure I'm doing something wrong in Regex part.

Answer 1

Your regex pattern is correct and does give the expected filter output. Refresh your index pattern in Kibana or try re-ingesting the data.

Although, I do not think you need to use regex if the channel pattern is going to be like Some Data:IP Address:Port

Try below pattern

grok{
      match => { "message" => ["%{TIMESTAMP_ISO8601:ts} %{LOGLEVEL:loglevel} %{WORD:Metric}: (?<Channel>%{DATA}:%{HOSTPORT}) (?<Data>%{GREEDYDATA})"]}
    }

Logstash output will be

{
            "ts" => "2020-11-06 12:57:43,854",
        "Metric" => "Bandwidth",
    "@timestamp" => 2020-11-06T22:47:20.383Z,
      "loglevel" => "INFO",
          "host" => "e7c15acec470",
          "Data" => "0.000059 Gb/S",
       "Channel" => "NASDAQ:224.0.130.65:30408",
      "@version" => "1",
       "message" => "2020-11-06 12:57:43,854 INFO Bandwidth: NASDAQ:224.0.130.65:30408 0.000059 Gb/S"
}

Try using stdout output along with elasticsearch so you can see what logstash is outputting to elastic.

output{
      stdout { codec => rubydebug }
    }