How to configure RabbitMQ with Filebeat

0
votes

We use ELK to control our logs and visualize them in Kibana. We have about 30 different log types, that we are sending both from our grid machines and both from our client side machines.

The client side machines work with filebeat-->logstash-->elastic-->Kibana

I want to change the config from:
filebeat-->logstash-->elastic-->Kibana
to:
filebeat-->Rabbitmq-->logstash-->elastic-->Kibana

In elastic I founded this yml examples - when RabbitMQ isn't one of them.
Is it possible to config our system in this way? or I have to use:
filebeat-->logstash1-->Rabbitmq-->logstash2-->elastic-->Kibana

2
elasticsearchrabbitmq
I don't know what your options are, but have you tried Kafka as a filebeat output? Logstash also support Kafka as an input. – Fabian76
In the end we stayed in the same configuration at first. Later on we started to change the project to Serilog with elastic as a sink – Green

2 Answers

3
votes

I'm not familiar with RabbitMQ in this context, but this seems to be an open issue in the Github repository for Beats:

https://github.com/elastic/beats/issues/581

There's also this post from the Elastic forums that indicates filebeat doesn't have plugin support, so I'm guessing the first option is no:

https://discuss.elastic.co/t/filebeat-rabbitmq-plugin/55189

Logstash certainly supports rabbitmq as an output:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-rabbitmq.html

1
votes

I found in the end a nice implementation (although it's not an "official" one) for a FileBeat output plugin. Because as wrote here correctly Beats team doesn't going to support RabbitMQ plugin someone wrote it himself.

In Summery, in order to use the plugin you should do the following:

  1. In the main.go file:
    package main

    import (
    "os"

    _ "github.com/sidleal/mqttout"

    "github.com/sidleal/countbeat/cmd"
)

func main() {
        if err := cmd.RootCmd.Execute(); err != nil {
            os.Exit(1)
        }
}

  2. In the Config filebeat (yourbeat.yml):
    config file

    output.mqtt:
    host: "127.0.0.1"
    port: 1883
    topic: "mytopic"
    user: "myvhost:myuser"
     password: "mypassword"

  3. bind amq.topic exchange to your desired queue, putting your topic in Routing Key.