Filebeat: Send logs from one Filebeat instance to different Logstash instances

Question

I am trying to get my Filebeat to deliver logs to multiple instances of Logstash. Filebeat can have multiple outputs, but they are then loadbalanced.

I have tried setting up filebeat with:

output.logstash.hosts: ["IP1:5044", "IP2:5044"]
output.logstash.loadbalance: false

Which gives me loadbalancing IP1 until that node fails, and then switches to IP2. Loadbalance = false, is defined as default when there are multiple hosts

Also, I have tried:

output.logstash.hosts: ["IP1:5044", "IP2:5044"]
output.logstash.loadbalance: true

Which gives me loadbalancing logs being sent. One log entry goes to IP1, and next one goes to IP2.

Aiming for completely redundant ELK pipelines where filebeat is supplying both at the same time.

Answer 1

What is the question here? Filebeat will always wait for an acknowledgement from Logstash and either method will retry if they try to send data to an unreachable instance.

The loadbalance attribute will only change if you send data to only one host (and switch on failure) or distribute amongst all of them.

You can only send each event to one Logstash instance; otherwise you'd have duplicate event in Elasticsearch in the end.