Rabbit MQ idle connection dropped

Question

I have a .NET windows service running as a consumer/subscriber which is listening to a queue for messages.

The windows service is running on the same machine as where rabbit mq server s/w is installed.

The queue if idle for 60 minutes results in the connection for it being dropped (i know this as i monitor the UI dashboard) and puts the windows service into a bad state.

This is proving to be frustrating to resolve. I have applied heart beat setting on the rabbit mq client but this has had no effect.

The following error is what i get in the log file when connection drops

=ERROR REPORT==== 22-Aug-2017::12:20:29 ===
closing AMQP connection <0.1186.0> ([FE80::C00E:F801:A2A7:8530]:61481 -> 
[FE80::C00E:F801:A2A7:8530]:5672):
missed heartbeats from client, timeout: 30s

Rbbit mq server log file settings: [{rabbit,[ {heartbeat, 60}]}].

Client code:

var connectionFactory = new ConnectionFactory
        {
            HostName = hostName,
            UserName = userName,
            Password = password,
            RequestedHeartbeat = heartBeat,
            AutomaticRecoveryEnabled = true,
            NetworkRecoveryInterval = TimeSpan.FromSeconds(numberOfSecondsInterval),
            RequestedConnectionTimeout = RequestedConnectionTimeoutInMiliseconds
        };

        if (port > 0)
            connectionFactory.Port = port;

        var connection = connectionFactory.CreateConnection();

        var model = connection.CreateModel();

        model.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false); 

        return new Tuple<IConnection, IModel>(connection, model);

heartbeat value above is set to 30 seconds,

network recovery value is set to 10 seconds &

request connection time out is set to 2 seconds

i don't know what else i'm missing here in terms of configuration??

The server where above is running from is Windows 2012 R2

Basically i'm expecting that to see the connections remain in place always regardless of idle time.

Is there a Windows OS level TCP keep-alive setting i need to make sure is in place as well?

Rabbit MQ version is 3.6.8

tearing my hair out on this one so any pointers greatly appreciated

Is there a firewall? they seem to interrupt idle connections aswell. — Samer Tufail
Windows firewall is turned off on the machine, the win service and rabbit mq server are on the same machine as well — AdrianSean
I had changed the server name to localhost instead of the machine name in the connection settings for the win service but that had no affect — AdrianSean
You could try: intercept heartbits using wireshark or similar (you will need to put your rabbitmq server on a network or VM). Check if the issue still occurs. If so, then you can figure out if the heartbits are really reaching the RabbitMQ server. Maybe your app actually stops sending them for some reason. Another thing: you will need to implement reconnection logic anyway, you should not trust the connection being there forever because your client and server can be partitioned one from another. Looks like your code is .NET, you could look at EasyNetQ or MassTransit — Alex Buyny
thanks Alex, reconnect logic looks like the way to go here as all other configurations dont resolve the issue — AdrianSean

AdrianSean AdrianSean · Accepted Answer · 2017-08-26T13:07:36

I managed to successfully stop the idle connections from dropping (after 60 mins) on RabbitMQ server by applying the re-connect logic that was referenced in this SO post.

To note: The answer was updated to state that the latest version of RabbitMQ client has auto connection recovery enabled so manual re-connection logic should not be needed. This was not true in my case as i had applied these settings already but i still saw the connections dropping after 60 minutes idle time. The client and the server in my scenario are on the same machine.

If anyone by any chance knows where the 60 minutes idle time setting is coming from i would be grateful, i scanned all the rabbitmq config settings and could not find anything related to it.

Rabbit MQ idle connection dropped

1 Answers