Error connecting rabbitmq cluster on Amazon EC2

Question

I am experiencing some difficulties connecting two RabbitMQ nodes on amazon EC2. The two nodes are controlled using puppet, here is my rabbit.config file:

[
  {mnesia, [{dump_log_write_threshold, 1000}]},
  {rabbit, [
        {tcp_listeners, [5672]},
            {kernel, [{inet_dist_listen_min, 55700},{inet_dist_listen_max, 55800}]} ,
        {cluster_nodes, ['rabbit@server1', 'rabbit@server2']}
        ]
    }
].

I believe the rights ports for the cluster to connect are open. I am able to telnet from server2 to server1 on both 5672 and 4369.

I have the same /var/lib/rabbitmq/.erlang.cookie on both servers.

And from erlang command line when I net_admin:ping the other node I get pang back.

However, when I run cluster_status on any node they do not look like they are aware of each other. Doing stop_app, reset,rabbitmqctl cluster rabbit@server1 I always get the following error: Error: {no_running_cluster_nodes...

Has anybody solved a similar problem, or know how to solve it?

Answer 1

Have you opened the ports between 55700 and 55800?

Try checking this to understand what other ports RabbitMQ listens on:

netstat -plten | grep beam

And I'd double-check the cookie...

Answer 2

Like Ivan suggests, you can check which ports the servers are listening on first and then add those TCP rules to Security Groups for servers. That's a good first step.

netstat -plten | grep beam

Returns the following (if server still running and not stop_app)

tcp        0      0 0.0.0.0:37419               0.0.0.0:*                   LISTEN      498        118739     15519/beam          
tcp        0      0 0.0.0.0:15672               0.0.0.0:*                   LISTEN      498        119032     15519/beam          
tcp        0      0 0.0.0.0:55672               0.0.0.0:*                   LISTEN      498        119029     15519/beam          
tcp        0      0 :::5672                     :::*                        LISTEN      498        119018     15519/beam

Notice the common ports 5672 15672 55672 for amqp and web server and the other port is the port the cluster is listening on. Check your other instances and make sure your range includes both of them, then retry and it will work.

Security Group > Inbound > TCP Rule: 30000-65535 and the Security Group allowed sg-XXXXXX and repeat for reciprocating security groups and don't forget to "Apply Rules".

2. Next make sure you share the /var/lib/rabbitmq/.erlang.cookie (just copy from one server to all others and restart instances)

3. Then on your command line:

   [root@ip-172-31-27-150 ~]# rabbitmqctl stop_app Stopping node 'rabbit@ip-172-31-27-150' ... ...done.

   [root@ip-172-31-27-150 ~]# rabbitmqctl reset Resetting node 'rabbit@ip-172-31-27-150' ... ...done.

   [root@ip-172-31-27-150 ~]# rabbitmqctl join_cluster rabbit@ip-172-31-28-79 Clustering node 'rabbit@ip-172-31-27-150' with 'rabbit@ip-172-31-28-79' ... ...done.

4. Lastly, don't forget to restart your instance rabbitmqctl start_app

This worked for me on 5 EC2 instances.

Answer 3

thanks for your answer, what I did is to remove the content of this directory except .erlang.cookie ( rm -R /var/lib/rabbitmq/ ). And the cluster connected successfully. Cheers!