ELK - How to use different source in logstash

Question

I have a so far running ELK installation that I want to use to analyse log files from differenct sources:

nginx-logs
auth-logs
and so on...

I am using filebeat to collect content from logfiles and sending it to logstash with this filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
     - /var/log/*.log
    - /var/nginx/example_com/logs/
output.logstash:
  hosts: ["localhost:5044"]

In logstash I alread configured a grok-section, but only for nginx-logs. This was the only working tutorial I found. So this config receives content from filebeat, filters is (that's what grok is for?) and sends it to elasticsearch.

input {
   beats {
      port => 5044
   }
}

filter {
   grok {
      patterns_dir => "/etc/logstash/patterns"
      match => { "message" => "%{NGINXACCESS}" }
   }
}

output {
   elasticsearch {
       hosts => "localhost:9200"
       manage_template => false
       index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
       document_type => "%{[@metadata][type]}"
   }
}

That's the content of the one nginx-pattern file I am referencing:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} (?:-|(%{WORD}.%{WORD})) %{USER:ident} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{QS:forwarder}

But I have trouble understanding how to manage different log-data sources. Because now Kibana only displays log content from /var/log, but there is no log data from my particular nginx folder.

What is it, that I am doing wrong here?

I don't understand the role of "filebeat" there. From my understanding, filebeat collects and reads the logfiles and send the content to logstash where I process them (filter / grok). But looks like as it is totally sufficient to just read logfiles with logstash, as described over there. — n.r.
@nr filebeat allow to have one logstash for multiples log sources on multiple machines (=> easier to manage). Also logstash consume a lot of resources, so installing logstash on another machine and using filebeat to send the logs can make sense. — baudsp
@baudsp Got it. So to summarize: Logstash could theoretically be configured to have multiple sources, but one should prefer filebeat and run logstash on a different, single machine. — n.r.

andre-lx andre-lx · Accepted Answer · 2019-01-14T14:36:14

Since you are running filebeat, you already have a module available, that process nginx logs filebeat nginx module

This way, you will not need logstash to process the logs, and you only have to point the output directly to elasticsearch.

But, since you are processing multiple paths with different logs, and because elastic stack don't allow to have multiple output forms (logstash + elasticserach), you can set logstash to only process logs that do not come from nginx. This way, and using the module (that comes with sample dashboards) , your logs will do: Filebeat -> Logstash (from input plugin to output plugin - without any filtering) -> Elasticsearch

If you really want to process the logs on your own, you are in a good path to finish. But right now, all your logs are being process by the grok pattern. So maybe the problem is with your pattern, that processes logs from nginx, and not from nginx in the same way. You can filter the logs in the filter plugin, with something like this:

#if you are using the module    
filter {
      if [fileset][module] == "nginx" {   

      }
 }

if not, please check different available examples at logstash docs

Another thing you can try, it's add this to you filter. This way, if the grok fails,you will see the log in kibana, but, with the "_grok_parse_error_nginx_error" failure tag.

grok {
      patterns_dir => "/etc/logstash/patterns"
      match => { "message" => "%{NGINXACCESS}" }
      tag_on_failure => [ "_grok_parse_error_nginx_error" ]
   }

ELK - How to use different source in logstash

1 Answers