Cannot Start RabbitMQ node after erlang cookie change

Question

I am new to RabbitMQ. Trying to make cluster of 3 nodes on RHEL 7 server. Need to make node3 as master and node1,node2 as its slaves. I copied .erlang.cookie content of master node to the slave nodes. Tried to make cluster but nodedown error msg comes up.

However,as a part of troubleshooting discussed in here previously, I killed rabbit process.

Stopped rabbitmq broker , Trying to start the broker but couldn't. Issued "systemctl start rabbitmq-server.service" and then issued "journalctl -xe" for more details on the error

Error Message :

**

Jan 29 03:56:37 vmrabbitmqt02 systemd[1]:
 Starting RabbitMQ broker...
-- Unit rabbitmq-server.service has begun starting up.
Jan 29 03:56:38 vmrabbitmqt02 systemd[1]: Cannot find unit for notify message of PID 5041.
Jan 29 03:56:38 vmrabbitmqt02 systemd[1]: Cannot find unit for notify message of PID 5043.
Jan 29 03:56:38 vmrabbitmqt02 rabbitmq-server[4839]: {"could not start kernel pid",application_controller,"error in config file \"/etc/rabbitmq/rabbitmq.config\" (5): s
Jan 29 03:56:39 vmrabbitmqt02 rabbitmq-server[4839]: Crash dump was written to: erl_crash.dump
Jan 29 03:56:39 vmrabbitmqt02 rabbitmq-server[4839]: could not start kernel pid (application_controller) (error in config file "/etc/rabbitmq/rabbitmq.config" (5): synt
Jan 29 03:56:39 vmrabbitmqt02 systemd[1]: rabbitmq-server.service: main process exited, code=exited, status=1/FAILURE
Jan 29 03:56:40 vmrabbitmqt02 systemd[1]: Cannot find unit for notify message of PID 5222.
Jan 29 03:56:40 vmrabbitmqt02 systemd[1]: Cannot find unit for notify message of PID 5223.
Jan 29 03:56:40 vmrabbitmqt02 systemd[1]: Cannot find unit for notify message of PID 5224.
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: Stopping and halting node rabbit@vmrabbitmqt02 ...
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: Error: unable to connect to node rabbit@vmrabbitmqt02: nodedown
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: DIAGNOSTICS
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: ===========
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: attempted to contact: [rabbit@vmrabbitmqt02]
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: rabbit@vmrabbitmqt02:
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: *** connected to epmd (port 4369) on vmrabbitmqt02
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: * epmd reports: node 'rabbit' not running at all
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: no other nodes on vmrabbitmqt02
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: * suggestion: start the node
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: current node details:**
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: - node name: 'rabbitmq-cli-51@vmrabbitmqt02'
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: - home dir: /var/lib/rabbitmq
Jan 29 03:56:40 vmrabbitmqt02 rabbitmqctl[5115]: - cookie hash: E8X5sOmIM3Y+Ju6lgvSIhA==
Jan 29 03:56:40 vmrabbitmqt02 systemd[1]: Failed to start RabbitMQ broker.

**

v

* epmd reports: node 'rabbit' not running at all

Snapshot of rabbitmq-env.conf file along with details of hosts Please suggest some measures at the earliest.

please post your /etc/rabbitmq/rabbitmq.config most likely you have an error there — Gabriele Santomaggio
Posted the config file. Can you please suggest what am I missing here ? — Shweta Lokhande
` (error in config file "/etc/rabbitmq/rabbitmq.config" (5):` I read this error — Gabriele Santomaggio
I placed the config file from old server and changed the server names in the config file. Now I can see nodedown error and it is not getting started. I issued systemctl enable rabbitmq-server.service system start rabbitmq-server.service rabbitmqctl start_app — Shweta Lokhande

Shweta Lokhande Shweta Lokhande · Accepted Answer · 2017-01-29T19:37:10

Config File used :

[root@vmrabbitmqt02 rabbitmq]# cat rabbitmq.config
% This file managed by Puppet
% Template Path: rabbitmq/templates/rabbitmq.config
[
{rabbit,
[{cluster_nodes, {['rabbit@vmrabbitmqt01', 'rabbit@vmrabbitmqt02', 'rabbit@vmrabbitmqt03'], disc}},
{cluster_partition_handling, ignore},
{ssl_listeners, [5671]},
{ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/rabbitmq_wildcard_root_ca.crt"},
                {certfile,"/etc/rabbitmq/ssl/rabbitmq_wildcard.crt"},
                {keyfile,"/etc/rabbitmq/ssl/rabbitmq_wildcard.key"},
                {verify,verify_none},
                {fail_if_no_peer_cert,false}]},
{default_user, <<"guest">>},
{default_pass, <<"guest">>}
]}
].
% EOF

I did an exit from root user. Logged in again .. Now Getting a node down error..

Stopping node rabbit@vmrabbitmqt02 ...

Error: unable to connect to node rabbit@vmrabbitmqt02: nodedown

DIAGNOSTICS

attempted to contact: [rabbit@vmrabbitmqt02]

rabbit@vmrabbitmqt02:

connected to epmd (port 4369) on vmrabbitmqt02
epmd reports node 'rabbit' running on port 25672
TCP connection succeeded but Erlang distribution failed
suggestion: hostname mismatch?
suggestion: is the cookie set correctly?
suggestion: is the Erlang distribution using TLS?

current node details:

node name: 'rabbitmq-cli-90@vmrabbitmqt02'
home dir: /var/lib/rabbitmq
cookie hash: xy56ebUJo+5exKrQIexXrg==

Cannot Start RabbitMQ node after erlang cookie change

1 Answers

DIAGNOSTICS