RabbitMQ Spring Boot application handle broker down

Question

I have a Spring Boot application configured to connect to a local RabbitMQ broker. I have also configured a FixedBackOff strategy in order to stop retrying connecting after 3 unsuccessful attempts.

EDIT 1: I have the following configuration for FixedBackOff as explained by @gary-russell in here:

@Bean(name = "rabbitListenerContainerFactory")
public SimpleRabbitListenerContainerFactory simpleRabbitListenerContainerFactory(
        SimpleRabbitListenerContainerFactoryConfigurer configurer,
        ConnectionFactory connectionFactory) {
    SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
    configurer.configure(factory, connectionFactory);
    BackOff recoveryBackOff = new FixedBackOff(5000, 3);
    factory.setRecoveryBackOff(recoveryBackOff);
    return factory;
}

With this configuration, I keep getting a ConnectionException in the console:

2018-04-16 13:48:29.769  WARN 54952 --- [nfoReplicator-0] o.s.b.a.health.RabbitHealthIndicator     : Health check failed

org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:368)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:573)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1430)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1411)
    at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1387)
    at org.springframework.boot.actuate.health.RabbitHealthIndicator.getVersion(RabbitHealthIndicator.java:49)
    at org.springframework.boot.actuate.health.RabbitHealthIndicator.doHealthCheck(RabbitHealthIndicator.java:45)
    at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:43)
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68)
    at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getHealthStatus(EurekaHealthCheckHandler.java:103)
    at org.springframework.cloud.netflix.eureka.EurekaHealthCheckHandler.getStatus(EurekaHealthCheckHandler.java:99)
    at com.netflix.discovery.DiscoveryClient.refreshInstanceInfo(DiscoveryClient.java:1362)
    at com.netflix.discovery.InstanceInfoReplicator.run(InstanceInfoReplicator.java:100)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:50)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:907)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:859)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:799)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:352)
    ... 19 common frames omitted

Therefore the application does not go up and keeps on printing this exception. How can I start this application even when the broker is down?

Answer 1

That SimpleRabbitListenerContainerFactoryConfigurer is not related to what you have in the logs.

The container is about @RabbitListener - even-driven consumers, but here you have a RabbitHealthIndicator which uses already a RabbitTemplate and it is a passive call, therefore no any retry is applied here. Well, you can configure some retry via spring.rabbitmq.template.retry configuration properties, but that won't help you to stay UP with that health indicator meanwhile you are really DOWN.

Answer 2

I made this way:

@Configuration
@EnableRabbit
public class AmqpConfiguration {

    @Bean
    public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
        SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
        factory.setConnectionFactory(connectionFactory);
        ExponentialBackOff recoveryBackOff = new ExponentialBackOff();
        factory.setRecoveryBackOff(recoveryBackOff);
        return factory;
    }
}

Using the BackOffExponencial class, your server will check whether rabbitMq is online. This service will check in exponential time and it will never stop.

Exemple:

For 10 attempts the sequence will be as follows:

request # ... back off

1 ................. 2000

2 ................. 3000

3 ................. 4500

4 ................. 6750

5 ................. 10125

6 ................. 15187

7 ................. 22780

8 ................. 30000

9 ................. 30000

10 ................. 30000