Logstash grok pattern for kafka logs

1
votes

I'm trying to ship Kafka logs to elk for which i'm using below grok filter in logstash:

grok {
match => {"actual_message" => "%{SYSLOG5424SD} %{LOGLEVEL} %{GREEDYDATA}"}
}

Kafka Log format:

[2019-09-24 12:54:11,546] DEBUG [Controller id=0] Topics not in preferred replica Map() (kafka.controller.KafkaController)
[2019-09-24 12:54:11,546] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)

When I use this, the 'timestamp' of actual logs in Kibana is not shown correctly. To clarify, if you notice the 'message' section in below image, the event occurred on 24th September but the "@timestamp" shows 26th September, the date and time when I started shipping logs to elk. Can anyone tell me the actual grok pattern which would fix the issueLogs in Kibana

1
logstash-grok
I think you don't mean 'doesn't show', 'shows' makes far more sense. If I am right, please edit the question accordingly. - MandyShaw
Thanks, i have edited the post. - Eva
You can mutate the timestamp field as mentioned here discuss.elastic.co/t/… - Sahadevan

1 Answers

0
votes

Try this:

\[%{TIMESTAMP_ISO8601:timestamp}] %{LOGLEVEL} %{GREEDYDATA}