Filebeat : Send different logs from filebeat to different logstash Pipeline

2
votes

I Want the functionality that one filebeat instance can send data to different logstash pipeline.

Is this possible?

I have configured one logstash service having two pipelines, both pipelines separate ports are given. Let's say Pipeline1 (Port 5044) , Pipeline2 (Port 5045)

Now i want to send data to the logstash using filebeat. So i have two types of log file let's say log1, log2.

I want to send log1 to Pipeline1 and log2 to Pipeline 2.

I am running only one instance of filebeat, how i can do this?

2
elasticsearchlogstashkibanafilebeat
Why you need two different logstash Pipelines? – ElasticCode

2 Answers

2
votes

Filebeat can have only one output, you will need to run another filebeat instance or change your logstash pipeline to listen in only one port and then filter the data based in tags, it is easier to filter on logstash than to have two instances.

In Filebeat you can specify a tag for each input that you have and use those tags in your logstash to send the log to desired pipeline.

For example, events with the tag log1 will be sent to the pipeline1 and events with the tag log2 will be sent to the pipeline2.

Your configuration needs to be something like this in Filebeat:

- type: log
  enabled: true
  paths:
    - "/path/to/your/logs/*.json"
  tags: ["logN"]

And then you will need a conditional in your logstash filters and outputs to each tag you want:

filter {
    if "logN" in [tags] {
        filters
    }
}
output {
    if "logN" in [tags] {
        output
    }
}
0
votes

Filebeat can have only one output, but this can be achived by using an messaging medium between filebeat and logstash, i am using kafka in my case between filebeat and logstash to achieve the above request.