Forwarding docker GELF logs to Logstash with Filebeat (or alternative?)

Question

Gelf messages are a subset of all Json Strings. How can I use filebeat (or an alternative) as a lightweight solution to forward docker gelf logs reliably to logstash?

Further info:

I have a cluster (docker swarm for now) of machines in the same network running docker containers. I want to use --log-driver=gelf because I like the gelf format and want the fields that docker adds to each GELF log entry.

Unfortunately docker sends GELF logs with UDP and I fear loosing log entries. Either because packages are lost, logstash is down, or there is too much load for logstash. I don't want to run logstash on each host because it is a heavyweight.

Answer 1

Try to place Rabbitmq or Redis in front of GELF.

You'll want to split the filtering from the ingestion in a centralized manner, add several Logstash shippers or just have a way to buffer new logs from any type of slower parsing. You can split the original log.conf into two files depending if they’re reading into Redis or grabbing from the queue, parsing and sending to ES.