Connecting Logstash and Rabbitmq docker containers

Question

I'm trying to launch Logstash from a docker image to send logs to a RabbitMQ instance running on another docker container.

I linked both containers and launched Logstash with the following command and config file:

docker run --link rabbitmq:rabbit -it log-sender logstash -f config.conf

config.conf:

input { 
    stdin {
    } 
}
output {
    rabbitmq {
        host => "RABBIT_PORT_5672_TCP_ADDR"
        exchange => "test_exchange"
        exchange_type => "fanout"
    }
}

I obtained the following error:

RabbitMQ connection error: Connection to RABBIT_PORT_5672_TCP_ADDR:5672 refused: host unknown. Will attempt to reconnect in 10 seconds... {:exception=>#, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare- 2.1.2-java/lib/march_hare/session.rb:361:in 'converting_rjc_exceptions_to_ruby'" , "/opt/logstash/vendor/bundle/jruby/1.9/gems/march_hare-2.1.2-java/lib/march_ha re/session.rb:382:in 'new_connection_impl'", "/opt/logstash/vendor/bundle/jruby/ 1.9/gems/march_hare-2.1.2-java/lib/march_hare/session.rb:82:in 'initialize'", "/ ... uts/rabbitmq/march_hare.rb:18:in 'register'", "org/jruby/RubyArray.java:1613:in 'each'", "/opt/logstash/lib/logstash/pipeline.rb:220:in 'outputworker'", "/opt/l ogstash/lib/logstash/pipeline.rb:152:in `start_outputs'"], :level=>:error}

The rabbit docker is the base image, run with this command:

docker run -d -p 5672:5672 -p 15672:15672 rabbitmq

It seems that for some reason my Logstash image can't access to the port of the RabbitMQ image. Using curl I can get the response "AMPQ" when used from the host on port 5672, but not from the Logstash image.

Is host RABBIT_PORT_5672_TCP_ADDR reachable? Probably you have to specify conatiner hostname (-h or --hostname) and use it in your config, or if your logstash also reside inside container then simple localhost will be fine. — pinepain

hackenfreude hackenfreude · Accepted Answer · 2015-08-01T23:36:41

I ran into the same problem. I got pointed down the right path with the help of How to reference environment variables in logstash configuration file?.

Remember that your file above is a static Logstash config file. So the string RABBIT_PORT_5672_TCP_ADDR that appears in the file never gets expanded by Logstash into the actual IP address contained in the environment variable $RABBIT_PORT_5672_TCP_ADDR.

How do you get the correct IP address into the Logstash config file? You don't.

Docker does some magic with /etc/hosts on the containers. If you have your multi-container environment set up correctly, you should see an entry for your Rabbit container's name in your Logstash container's hosts file.

This allows you to use the Rabbit container's name as the address in Logstash config on the Logstash container:

rabbitmq {
    host => "name_of_your_rabbit_container"
}

If you want to double-check, copy the Logstash container's hosts file to your system. Open it up, and you should see an entry with a Docker-generated IP address and the name of your Rabbit container.

Connecting Logstash and Rabbitmq docker containers

1 Answers