How to manage input from multiple beats to centralized Logstash

Question

I want to use Elastic Stack for log aggregation for fetching logs from 10 machines. I wish to install Filebeat on 10 machines & grab the logs from each machine and send it to a centralized Logstash server which is installed in a separate machine. In separate machine, Logstash Elasticsearch & Kibana is installed. I require Logstash as I want to do processing & parsing of data after gathering the logs using beats.

As per this architecture, I am facing some issues of identifying and parsing the logs. How to make logstash identify to collect logs from multiple beats server's at once? Can i specify multiple host in logstash-beats plugin so that logstash will parse all the logs from 10 machines at once?

Should i define separate document_type in all the 10 machines as part of Filebeat Configuration which can be later leveraged in Logstash so that I define multiple types (using wildcard - tomcat*) in filter plugin.

Sample Filebeat Configuration for Single Machine Setup:-

################### Filebeat Configuration Example #########################
############################# Filebeat ####################################
filebeat:
  prospectors:
    -
      paths:
        - /location/to/file/catalina.out
      document_type: tomcat1
      scan_frequency: 5s
      input_type: log

output:
  logstash:
    hosts: ["<host-of-the-machine-on-which-logstash-is-installed>:5044"]
  console:
    pretty: true
  shipper:
  logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

This type of setup will be done on all 10 machines wherein value of document_type will only change.

Sample Logstash Configuration for Single Machine:-

input {
    beats   {
        host => "ip/of/machine/1"
        port => 5044
    }
}
filter {
    ........................
    ........................
    ........................
}
output{
    elasticsearch {
        hosts => "localhost:9200"
        index => "logs-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}

More ideas are welcome.

alpert alpert · Accepted Answer · 2016-08-15T14:24:21

Actually the host parameter for beats input plugin means The IP address to listen on. This is not the ip for filebat and actually not required at all.

So just specifying port to listen must be enough and I think the configuration you show will work. Logstash will listen logs from all 10 machines and will process them.

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-host

How to manage input from multiple beats to centralized Logstash

1 Answers