FileBeat not load balancing to multiple logstash (indexer) servers

1
votes

I tried load balancing with 2 different logstash indexer servers, but when I add, say 1000 lines to my log, filebeats sends logs exclusively to only one server (I enabled stdout and can visually check output too see which logstash server is receiving the log events)

My filebeats conf:

filebeat:
  prospectors:
    -
      paths:
        - "D:/ApacheLogs/Test/access.*.log"
      input_type: log
      document_type: my_test_log
      scan_frequency: 1s
  registry_file: "C:/ProgramData/filebeat/registry"
output:
  logstash:
    hosts: ["10.231.2.223:5044","10.231.4.143:5044"]
    loadbalance: true
shipper:
logging:
  files:

will there be added support to disable persistent TCP connection on filebeats? I currently cannot use AWS ELB since due to sticky connection it always sends to one logstash server until it gets reset. Is this not the right architecture for it? Should I be sending to redis queue instead? In filebeats I have no idea nor could find any documentation how to send to redis queue?

Something like these did not work, I can't even find a way to debug it because filebeats leaves no logs

filebeat:
  prospectors:
    -
      paths:
        - "D:/ApacheLogs/Test/access.*.log"
      input_type: log
      document_type: my_test_log
      scan_frequency: 1s
  registry_file: "C:/ProgramData/filebeat/registry"
output:
  redis:
    # Set the host and port where to find Redis.
    host: "logstash-redis.abcde.0001.usw2.cache.amazonaws.com"
    port: 6379
shipper:
logging:
  level: warning
  # enable file rotation with default configuration
  to_files: true
  files:
    path: C:\temp\filebeat.log

  • Version: On windows server: FileBeat (Windows - filebeat version 1.2.2 (386)) On logstash indexer server: logstash 2.3.2

  • Operating System: Windows server: Microsoft Windows NT 6.0.6002 Service Pack 2 Logstash indexer server: RHEL Linux 4.1.13-19.30.amzn1.x86_64

2
logstashload-balancingfilebeat
You can use -e to log to stderr: elastic.co/guide/en/beats/filebeat/current/… – alpert

2 Answers

1
votes

Filebeat should really solve this, but since they advertise it as being as lightweight as possible, don't hold your breath.

I don't know how easy it is to get HAProxy running on Windows, but it should solve your problem if you can get it installed:

https://serverfault.com/questions/95427/windows-replacement-for-haproxy

Use Layer4 roundrobin load balancing. You'll probably want to install an HAProxy on every machine with Filebeat. 1 HAProxy frontend will listen on localhost:5044, and it will map to multiple Logstash backends.

0
votes

You can send your filebeat output to redis via below config:

output:
   redis:
     host: "host"
     port: <port>
     save_topology: true
     index: "index-name"
     db: 0
     db_topology: 1
     timeout: 5
     reconnect_interval: 1