Spring AMQP client auto reconnect but do not consuming resume

Question

If We are using spring amqp client which automatically disable amqp client autorecovery properties and it using own autorecovery mechanism.So if i restart one of the cluster node which behind load balancer default spring amqp autorecovery did not work for the cluster, because amqp connection and channel already connected node is restarted spring amqp do not recovery and reconnect again different node.But if there are already connected and consuming queue remain node did not effect that restart node.

For example Our RabbitMQ cluster is consist of three node behind ELB on AWS inside autoscaling group using aws_peer_discovery plugin.Our Cluster policy is / HA Policy .* all {"ha-mode":"all","ha-sync-mode":"automatic","queue-master-locator":"random","queue-mode":"lazy"} 0

Our consumer client running on ECS with 4 task that already connected and consuming our durable and mirrored queue.That means there is only one queue that is durable and mirrored. consume with 4 consumer task.

Out consumer using default Spring AMQP CachingConnectionFactory as following code block

@Bean
public CachingConnectionFactory cachingConnectionFactory(ConnectionFactory connectionFactory) {
    return new CachingConnectionFactory(connectionFactory);
}

@Bean
public CachingConnectionFactory cachingConnectionFactory(ConnectionFactory connectionFactory) {
    return new CachingConnectionFactory(connectionFactory);
}

In that situation when a restart a node (with command line systemctl restart rabbitmq) in there consumers which are connected to this restarted node do not autorecovery properly i saw attempt to connect to the cluster and already connected but did not consume so when i execute rabbitmqctl list_connection show me all connection fine but rabbitmqctl list_consumers show me an absent consumers that has been connected to the restarter node before and restart node up again but consumer did not reconnect another node or the same node and did not consume current queue.

If I change spring ampq client connection to following line that means saying spring ampq no we are using default amqp-client autorecovery mechanisim do not using spring itself and Test same scenario again at this time we do not get same error so when i restart node test our consumer reconnect another node and continueu to consuming.

@Bean
public CachingConnectionFactory cachingConnectionFactory(ConnectionFactory connectionFactory) {
    CachingConnectionFactory cachingConnectionFactory =
            new CachingConnectionFactory(connectionFactory);
    cachingConnectionFactory.getRabbitConnectionFactory().setAutomaticRecoveryEnabled(true);
    return cachingConnectionFactory;

Why spring-amqp did not work properly.I am wondering if we are using cluster behind load balancer we have to using defalt ampq-client autorecovery mechanism or our spring-amqp client code block wrong or absent ??

As I said in the rabbitmq-users google group; you need to look at the logs for the application that is not behaving as you expect; if you can't figure it out from that, post the logs somewhere such as a github gist or on pastebin. — Gary Russell

Purija Beigi Purija Beigi · Accepted Answer · 2019-07-31T14:12:14

You could configure a Recovery BackOff which can be applied in a listenerContainer configuration. I would recommend using exponential backoff. For publisher interactions, it is a backoff-policy configuration of the rabbitTemplate.

documentation can be found in section 3.1.17. Message Listener Container Configuration

Spring AMQP client auto reconnect but do not consuming resume

1 Answers