Logstash not adding fields

Question

I am using Logstash 1.4.2 and I have the following conf file.

I would expect to see in Kibana in the "Fields" section on the left the options for "received_at" and "received_from" and "description", but I don't.

I see

• @timestamp
• @version
• _id
• _index
• _type host path

I do see in the _source section on the right side the following...

received_at:2015-05-11 14:19:40 UTC received_from:PGP02 descriptionError1!

So home come these don't appear in the list of "Popular Fields"?

I'd like to filter the right side to not show EVERY field in the _source section on the right. Excuse the redaction blocks.

input
{
    file {
        path => "C:/ServerErrlogs/office-log.txt"
        start_position => "beginning"
        sincedb_path => "c:/tools/logstash-1.4.2/office-log.sincedb"
        tags =>  ["product_qa", "office"]
    }
    file {
        path => "C:/ServerErrlogs/dis-log.txt"
        start_position => "beginning"
        sincedb_path => "c:/tools/logstash-1.4.2/dis-log.sincedb"
        tags =>  ["product_qa", "dist"]
    }   

}
filter {

    grok {
        match => ["path","%{GREEDYDATA}/%{GREEDYDATA:filename}\.log"]
        match => [ "message", "%{TIMESTAMP_ISO8601:logdate}: %{LOGLEVEL:loglevel} (?<logmessage>.*)" ]
        add_field => [ "received_at", "%{@timestamp}" ]
        add_field => [ "received_from", "%{host}" ]     
    }
    date {
        match => [ "logdate", "ISO8601", "yyyy-MM-dd HH:mm:ss,SSSSSSSSS" ]
    }
    #logdate is now parsed into timestamp, remove original log message too
    mutate {
        remove_field => ['message', 'logdate' ]
        add_field => [ "description", "Error1!" ]
    }
}

output {
  elasticsearch {          
    protocol => "http"
    host => "0.0.0.x"
  }
}

---

Update:

I have tired searching with a query like:

tags: data AND loglevel : INFO

then saving this query, and then reloading the page.

But still I don't see loglevel appearing as 'Popular Fields'

Answer 1

If the fields don't appear on the left side, it's probably a kibana caching problem. Go to Settings->Indices, select your index, and click the orange Refresh button.

Answer 2

I had the same issue with logstash not adding fields and after quite a lot of searching and testing other things, suddenly I had the solution (but I´am using the logstash-logback-encoder, so I have JSON already - if you don´t, then you need to transform things into JSON in the logstash "input"-phase).

I added a "json" plugin-filter, that did the magic for me:

filter {
    json {
        source => "message"
    }
}