Grok filter not working even though it works in grok debugger

Question

I'm using logstash-1.4.0 with elasticsearch 1.3.4 and kibana 3.1.1 (I know I'm outdated, that's the best I can do right now).

Log Example:

2016-05-31 16:05:33 RequestManager [INFO] The manual flag LOLROFLin TRALALA 123456Was changed to true

My grok filter:

filter {
    grok {
        match => { "message" => "%{DATESTAMP:timestamp} %{WORD:clazz} %{NOTSPACE:level} %{GREEDYDATA:content}"}
    }

    if (!([stack_trace])) and (!([clazz] == "RequestAsset")) {
        drop {}
    }
}

My questions are:

Why do I not see the grok fields in kibana? I only see the default fields but not mine. Grok Debugger shows success, but kibana does not work.
My goal is to drop any log message that does not have a stack trace OR is not from class (called clazz in my grok filter) "RequestAsset". Should this work? can I use the fields created by the grok filter in a seperate if filter?

EDIT: I realised what went wrong, I was using the log4j plugin which already seperates the log to its contents, and the field message was already just the message itself.

Jeroen Vandevelde Jeroen Vandevelde · Accepted Answer · 2016-05-31T14:26:41

I tested your grok filter in this grok debugger and it failed. So i have rewritten it.

Here is the correct grok filter.

filter {
grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:clazz} %{NOTSPACE:level} %{GREEDYDATA:content}"}
}

if (!([stack_trace])) and (!([clazz] == "RequestAsset")) {
    drop {}
}

TIMESTAMP_ISO8601 => %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

If you see "_grokparsefailure" in Kibana, you know that your grok filter failed.

On your second question shouldn't you use the OR operator?

Grok filter not working even though it works in grok debugger

2 Answers