Spring AMQP RabbitListener adjust auto reconnect configuration

Question

I'm trying to better understand the RabbitListener behavior when the broker goes down & when its brought back up. I have setup a rabbit in a docker container over localhost:6672 and created a simple spring boot application to listen for messages. Also configured application.yml to point to this rabbit.

        @RabbitListener(containerFactory = "myListenerContainerFactory", 
                bindings = @QueueBinding(value = @Queue(value = "MY_QUEUE"), exchange = @Exchange(value = "MY_EXCHANGE"), key = "MY_KEY"))
        public void onMessage(final Mesg message) {
            //some handling
        }

Also defined the below container factory

    @Bean
    public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory cf) {
        SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
        factory.setConnectionFactory(cf);
        factory.setMessageConverter(new Jackson2JsonMessageConverter());
        factory.setDefaultRequeueRejected(false);
        factory.setConsumerTagStrategy(q -> "My App");
        return factory;
    }

I brought the spring boot app up when the rabbit container was running & it was able to auto create the exchange/queue & able to consume messages that i posted through rabbit admin console. I removed the rabbit container forcefully (docker rm my_instance -f) while the spring boot app was running and it started printing the below message.

2020-05-26 17:03:15.097 ERROR 26816 --- [ 127.0.0.1:6672]  o.s.a.r.c.CachingConnectionFactory       : Channel shutdown: connection error
2020-05-26 17:03:15.614  INFO 26816 --- [cTaskExecutor-1]  o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer@69176296: tags=[[My App]], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:6672/,1), conn: Proxy@522c56e8 Shared Rabbit Connection: SimpleConnection@7578dfd0 [delegate=amqp://[email protected]:6672/, localPort= 65342], acknowledgeMode=AUTO local queue size=0
2020-05-26 17:03:15.618  INFO 26816 --- [cTaskExecutor-2]  o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: localhost:6672
2020-05-26 17:03:22.662  WARN 26816 --- [cTaskExecutor-2]  o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
2020-05-26 17:03:22.663  INFO 26816 --- [cTaskExecutor-2]  o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer@31978e0c: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-05-26 17:03:22.663  INFO 26816 --- [cTaskExecutor-3]  o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: localhost:6672
2020-05-26 17:03:29.683  WARN 26816 --- [cTaskExecutor-3]  o.s.a.r.l.SimpleMessageListenerContainer : Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect

After starting the rabbit container again, spring boot app seems to have detected it however it printed the below message & i don't think it has established a successful connection. Looking at the logs it seems to have 3 retries setup by default, is there a config that i can use to increase the limit?

Is there any best practice around handling such cases? My app is expected to connect with different rabbit clusters and its possible that 1 or the other may go on maintenance & i needed a way to re-connect automatically as soon as the rabbit cluster is back up.

2020-05-26 17:05:18.303  INFO 26816 --- [TaskExecutor-19]  o.s.a.r.l.SimpleMessageListenerContainer : Restarting Consumer@5053bc3c: tags=[[]], channel=null, acknowledgeMode=AUTO local queue size=0
2020-05-26 17:05:18.306  INFO 26816 --- [TaskExecutor-20]  o.s.a.r.c.CachingConnectionFactory       : Attempting to connect to: localhost:6672
2020-05-26 17:05:18.348  INFO 26816 --- [TaskExecutor-20]  o.s.a.r.c.CachingConnectionFactory       : Created new connection: myConnectionFactory#ebfe707:19/SimpleConnection@19d03268 [delegate=amqp://[email protected]:6672/, localPort= 49260]
2020-05-26 17:05:18.396  WARN 26816 --- [TaskExecutor-20]  o.s.a.r.listener.BlockingQueueConsumer   : Failed to declare queue: MY_QUEUE
2020-05-26 17:05:18.402  WARN 26816 --- [TaskExecutor-20]  o.s.a.r.listener.BlockingQueueConsumer   : Queue declaration failed; retries left=3

org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[MY_QUEUE]
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:710) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.passiveDeclarations(BlockingQueueConsumer.java:594) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:581) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.initialize(SimpleMessageListenerContainer.java:1196) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1041) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.io.IOException: null
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:1006) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:52) ~[amqp-client-5.4.3.jar:5.4.3]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1110) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    at com.sun.proxy.$Proxy283.queueDeclarePassive(Unknown Source) ~[na:na]
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:689) ~[spring-rabbit-2.1.4.RELEASE.jar:2.1.4.RELEASE]
    ... 5 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_QUEUE' in vhost '/', class-id=50, method-id=10)
    at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:288) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:138) ~[amqp-client-5.4.3.jar:5.4.3]
    ... 14 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_QUEUE' in vhost '/', class-id=50, method-id=10)
    at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:516) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:670) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:48) ~[amqp-client-5.4.3.jar:5.4.3]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:597) ~[amqp-client-5.4.3.jar:5.4.3]
    ... 1 common frames omitted

2020-05-26 17:05:23.420  WARN 26816 --- [TaskExecutor-20]  o.s.a.r.listener.BlockingQueueConsumer   : Failed to declare queue: MY_QUEUE
2020-05-26 17:05:23.425  WARN 26816 --- [TaskExecutor-20]  o.s.a.r.listener.BlockingQueueConsumer   : Queue declaration failed; retries left=2

Artem Bilan Artem Bilan · Accepted Answer · 2020-05-27T14:40:51

 reply-text=NOT_FOUND - no queue 'MY_QUEUE' in vhost '/'

You have a MY_QUEUE which is declared as auto-delete and looks like there is no its recreation during connection establishing. Consider to have it as a bean and let RabbitAdmin to handle its creation on the reconnection.

See more info in docs: https://docs.spring.io/spring-amqp/docs/2.2.7.RELEASE/reference/html/#broker-configuration

Spring AMQP RabbitListener adjust auto reconnect configuration

1 Answers