Elasticsearch Logstash Filebeat mapping

Question

Im having a problem with ELK Stack + Filebeat.

Filebeat is sending apache-like logs to Logstash, which should be parsing the lines. Elasticsearch should be storing the split data in fields so i can visualize them using Kibana.

Problem: Elasticsearch recieves the logs but stores them in a single "message" field.

Desired solution:

Input:

10.0.0.1 some.hostname.at - [27/Jun/2017:23:59:59 +0200]

ES:

"ip":"10.0.0.1"

"hostname":"some.hostname.at"

"timestamp":"27/Jun/2017:23:59:59 +0200"

My logstash configuration:

    input {

  beats {
    port => 5044
  }



}

filter { 


  if [type] == "web-apache" {
        grok {
            patterns_dir => ["./patterns"]
            match => { "message" => "IP: %{IPV4:client_ip}, Hostname: %{HOSTNAME:hostname}, - \[timestamp: %{HTTPDATE:timestamp}\]" }
            break_on_match => false
            remove_field => [ "message" ]
        }

        date {
            locale => "en"
            timezone => "Europe/Vienna"
            match =>  [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
        }

        useragent {
            source => "agent"
            prefix => "browser_"
        }
    }

    }

output {

    stdout {
       codec => rubydebug
    }

    elasticsearch {
             hosts => ["localhost:9200"]
         index => "test1"
             document_type => "accessAPI"
           }
}

My Elasticsearch discover output:

I hope there are any ELK experts around that can help me. Thank you in advance, Matthias

PtX PtX · Accepted Answer · 2017-08-31T17:33:06

The grok filter you stated will not work here.

Try using:

%{IPV4:client_ip} %{HOSTNAME:hostname} - \[%{HTTPDATE:timestamp}\]

There is no need to specify desired names seperately in front of the field names (you're not trying to format the message here, but to extract seperate fields), just stating the field name in brackets after the ':' will lead to the result you want.

Also, use the overwrite-function instead of remove_field for message. More information here:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-options

It will look similar to that in the end:

filter {
  grok {
    match => { "message" => "%{IPV4:client_ip} %{HOSTNAME:hostname} - \[%{HTTPDATE:timestamp}\]" }
    overwrite => [ "message" ]
  }
}

You can test grok filters here:
http://grokconstructor.appspot.com/do/match

Elasticsearch Logstash Filebeat mapping

1 Answers