1
votes

I have 10 servers that i have Filebeat installed in. Each server monitors 2 applications, a total of 20 applications.

I have one Logstash server which collects all the above logs and passes it to Elasticsearch after filtering of these logs.

To read one file from one server, I use the below Logstash configuration:

input {
  beats {
    port => 5044
  }
}
filter {
    grok {
match => {"message" =>"\[%{TIMESTAMP_ISO8601:timestamp}\]%{SPACE}\[%{DATA:Severity}\]%{SPACE}\[%{DATA:Plugin}\]%{SPACE}\[%{DATA:Servername}\](?<short_message>(.|\r|\n)*)"}
    }
} 
output {
  elasticsearch {
    hosts => ["<ESserverip>:9200"]
    index => "groklogs"
}
          stdout { codec => rubydebug }
}

And this is the filebeat configuration:

paths:
    - D:\ELK 7.1.0\elasticsearch-7.1.0-windows-x86_64\elasticsearch-7.1.0\logs\*.log

output.logstash:
  hosts: ["<logstaship>:5044"]

Can anyone please give me an example of

  1. How i should convert the above to receive from multiple applications from multiple servers.
  2. Should i configure multiple ports? How?
  3. How should i use multiple Groks?
  4. How can i optimize it in a single or minimal logstash configuration files?

How will a typical set up look. Please help me.

1

1 Answers

4
votes

You can use tags in order to differentiate between applications (logs patterns).
As Filebeat provides metadata, the field beat.name will give you the ability to filter the server(s) you want.

Multiple inputs of type log and for each one a different tag should be sufficient.
See these examples in order to help you.

Logstash

filter {
    if "APP1" in [tags] {
        grok {
             ...
        }
    }
    if "APP2" in [tags] {
        grok {
             ...
        }
    }
}

Filebeat

filebeat.inputs:
- type: log 
  paths:
    - /var/log/system.log
    - /var/log/wifi.log
  tags: ["APP1"]
- type: log 
  paths:
    - "/var/log/apache2/*"
  tags: ["APP2"]