ELK Stack - Elasticsearch index creation (logstash)

1
votes

I'm experimenting with ELK to analyze our log files. Following the available documentation, managed to set up the stack in my pc. Now I'm facing an issue with the elastic search index creation. Previously I was using filebeat -> logstash -> elasticsearch -> kibana combination and using the following logstash.conf file was able to send data to elasticsearch

input {
  beats {
   port => 5044
   type => "log"
  }
}

output {
  elasticsearch {
   hosts => "localhost:9200"
   manage_template => false
   index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   document_type => "%{[@metadata][type]}"
  }
}

And the index in elastic search was evaluated to

 "filebeat-*"

from the expression

  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

Now I changed the logstash.conf to based on my actual logfile

input { 

file
{
    path => "C:\logs\application.log"
    start_position => "beginning"
    codec => 
    multiline {
      charset => "ISO-8859-1"
      pattern => "^%{TIMESTAMP_ISO8601}"
      max_lines => 1000
      negate => true
      what => "previous"
    } 
}

}

filter {
 mutate {
     gsub => [ "message", "\r", "" ]
   }

   grok {
    patterns_dir => "./patterns"
    match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL1:loglevel} %{THREAD:thread} %{IP5:remoteipaddress} %{JAVA:logclass} %{GREEDYDATA:details}"}
  add_field => [ "received_at", "%{@timestamp}" ]
  add_field => [ "received_from", "%{host}" ]

  }
   date {
            match => [ "timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]
            remove_field => [ "timestamp" ]
        }


}

output {

  elasticsearch { 
                hosts => "localhost:9200"
    manage_template => false
     index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
     document_type => "%{[@metadata][type]}"
                }
                file {
     path => "C:\logs\output.txt"
  }
}

In this case, logstash is happy with the conf file, but the index I suggested for elastic search is not being evaluated properly.

If I inspect elastic search using the head plugin,

http://localhost:9200/_plugin/head/

The index appears as

%{[@metadata][beat]}-

enter image description here

I'm not sure why the index expression is not being evaluated now. Any pointers to solve this issue would be helpful.

Thanks in advance, San

2
elasticsearchlogstashkibanalogstash-groklogstash-configuration

2 Answers

1
votes

Certain logstash plugins utilise metadata to transfer fields you don't want to store in the document. In your first example, the beats input is setting certain metadata that's used later in the elasticsearch output to set the index and type. As the file input doesn't set these metadata fields, logstash will output the variable name instead of a blank string, hence why it sets an index of "%{[@metadata][beat]}-2016.04.05", the date is known, but the metadata field beat is not.

If you just leave the elasticsearch output as it's defaults it should work fine:

elasticsearch { hosts => "localhost:9200" }

If you leave manage_template as false, it'll also not apply the logstash- template and the field mappings may be a bit off, so I'd recommend leaving that as the default (true) again.

1
votes

Since you know what the index should be called, just put it in the elasticsearch outoput:

Since you know what the index should be called, just put it in the `elasticsearch` output:

output {
  elasticsearch { 
    hosts => "localhost:9200"
    manage_template => false
    index => "filebeat-%{+YYYY.MM.dd}"
    document_type => "whatever_type_filebeat_put_in"
  }
}