How to prevent fail fast on external connection failure like Redis server in Spring boot?

Question

Is there a way to prevent spring-boot application from failing on startup due to external connection failures? I have found other similar questions that suggest using @Lazy annotation to prevent @Configuration beans initialisation but this solution did not work for me with Spring Data Redis using Jedis client.

Also, other solutions like this one are specific to dependencies being used in the app. For example, Spring Cloud has below property to control failfast behaviour -

spring.cloud.config.fail-fast=true

You can use this project that I created for my problem to reproduce by shutting down the redis server.

Below is how my code looks like -

@Lazy
@Configuration
public class RedisConfiguration {
    @Value("${spring.redis.sentinel.master}")
    private String SENTINEL_MASTER;

    @Value("${spring.redis.sentinel.nodes}")
    private String SENTINEL_NODES;

    @Value("${spring.redis.security.enabled:false}")
    private boolean REDIS_SECURITY_ENABLED;

    @Value("${spring.redis.security.password:}")
    private String REDIS_PASSWORD;

    @Lazy
    @Bean // somehow this always gets initialized
    public RedisConnectionFactory jedisConnectionFactory() { 
        // create set of sentinel nodes
        System.out.println(SENTINEL_NODES);
        Set<String> sentinelNodesSet = new HashSet<>(5);
        StringTokenizer st = new StringTokenizer(SENTINEL_NODES, ",");
        while (st.hasMoreTokens())
            sentinelNodesSet.add(st.nextToken());
        RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration(SENTINEL_MASTER, sentinelNodesSet);
        if (REDIS_SECURITY_ENABLED) {
            sentinelConfig.setPassword(REDIS_PASSWORD);
        }
        JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory(sentinelConfig);
        return jedisConnectionFactory;
    }

Below is the exception trace -

org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'stringRedisTemplate' defined in class path resource [org/springframework/boot/autoconfigure/data/redis/RedisAutoConfiguration.class]: Unsatisfied dependency expressed through method 'stringRedisTemplate' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jedisConnectionFactory' defined in class path resource [com/springboot/redisintegration/RedisConfiguration.class]: Invocation of init method failed; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is mysentinel master is running... at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:797) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:538) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1176) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:556) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) ~[spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.3.RELEASE.jar:2.3.3.RELEASE] at com.springboot.redisintegration.RedisIntegrationApplication.main(RedisIntegrationApplication.java:21) ~[classes/:na] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jedisConnectionFactory' defined in class path resource [com/springboot/redisintegration/RedisConfiguration.class]: Invocation of init method failed; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is mysentinel master is running... at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1794) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:226) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:884) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:788) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] ... 20 common frames omitted Caused by: redis.clients.jedis.exceptions.JedisConnectionException: All sentinels down, cannot determine where is mysentinel master is running... at redis.clients.jedis.JedisSentinelPool.initSentinels(JedisSentinelPool.java:249) ~[jedis-3.3.0.jar:na] at redis.clients.jedis.JedisSentinelPool.(JedisSentinelPool.java:154) ~[jedis-3.3.0.jar:na] at redis.clients.jedis.JedisSentinelPool.(JedisSentinelPool.java:122) ~[jedis-3.3.0.jar:na] at redis.clients.jedis.JedisSentinelPool.(JedisSentinelPool.java:116) ~[jedis-3.3.0.jar:na] at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createRedisSentinelPool(JedisConnectionFactory.java:374) ~[spring-data-redis-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.createPool(JedisConnectionFactory.java:358) ~[spring-data-redis-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.afterPropertiesSet(JedisConnectionFactory.java:342) ~[spring-data-redis-2.3.3.RELEASE.jar:2.3.3.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1853) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1790) ~[spring-beans-5.2.8.RELEASE.jar:5.2.8.RELEASE] ... 31 common frames omitted

In short:

@Lazy annotation works for RedisStandaloneConfiguration but not RedisSentinelConfiguration, not sure why?
Using @Lazy annotation is risky because you need to make sure all your services which are using Redis are loaded lazily too.
Looking for a solution like spring.cloud.config.fail-fast=true provided for spring cloud.

Update:

I've created below Jira issue for this feature -

https://jira.spring.io/browse/DATAREDIS-1208

jccampanero jccampanero · Accepted Answer · 2020-08-26T22:40:41

Under the hood Spring Cloud Config uses spring-retry and AOP (spring-boot-starter-aop) to configure a retry mechanism.

This process is implemented in ConfigServiceBootstrapConfiguration.

The relevant part of the code is this:

/* @ConditionalOnProperty("spring.cloud.config.fail-fast") */
@ConditionalOnClass({ Retryable.class, Aspect.class, AopAutoConfiguration.class })
@Configuration(proxyBeanMethods = false)
@EnableRetry(proxyTargetClass = true)
@Import(AopAutoConfiguration.class)
@EnableConfigurationProperties(RetryProperties.class)
protected static class RetryConfiguration {

  @Bean
  @ConditionalOnMissingBean(name = "configServerRetryInterceptor")
  public RetryOperationsInterceptor configServerRetryInterceptor(
      RetryProperties properties) {
    return RetryInterceptorBuilder.stateless()
        .backOffOptions(properties.getInitialInterval(),
            properties.getMultiplier(), properties.getMaxInterval())
        .maxAttempts(properties.getMaxAttempts()).build();
  }

}

As you can see, the basic idea is to provide a RetryConfiguration that handles a certain number of retries before considering the application failed.

The documentation of Spring Cloud Client provide more information about the different properties used for configuring this mechanism. You can see the default values also in the source code of the RetryProperties class.

Please, try and include the two required dependencies, spring-retry and spring-boot-starter-aop, and the RetryConfiguration as a child of your main configuration, provide some reasonable defaults for the configuration properties of the retry mechanism, and see what happens.

You can push the solution to the limit and try to reconnect on a very large number of occasions, perhaps increasing the cadence between them, waiting for the server to be available.

I think that the @Lazy annotations will no be longer necessary and could be safely removed.

EDIT

Reviewing your error stack trace, one think you can also try is to disable String Boot Redis auto-configuration classes.

You can do it in your annotations:

@SpringBootApplication(
  exclude = { RedisAutoConfiguration.class, RedisRepositoriesAutoConfiguration.class }
)

Or in your properties files:

spring.autoconfigure.exclude= \
org.springframework.boot.autoconfigure.data.redis.RedisAutoConfiguration, \
org.springframework.boot.autoconfigure.data.redis.RedisRepositoriesAutoConfiguration

You can disable Redis repositories configuration with this property also:

spring.data.redis.repositories.enabled: false

Once you disable the Redis auto-configuration, you will be free to instantiate the RedisTemplate or the stuff you need to interact with Redis when you consider appropriate.

You can, for instance, initialize it on demand, trying to establish a connection to Redis, by initializing all the required factories, when it is actually required. You can surround with try and catch the logic required to initialize your Redis connection and only initialize the RedisTemplate if a connection is available, something like the following.

On one hand:

public RedisConnectionFactory jedisConnectionFactory() { 
    try {
      // create set of sentinel nodes
      System.out.println(SENTINEL_NODES);
      Set<String> sentinelNodesSet = new HashSet<>(5);
      StringTokenizer st = new StringTokenizer(SENTINEL_NODES, ",");
      while (st.hasMoreTokens())
          sentinelNodesSet.add(st.nextToken());
      RedisSentinelConfiguration sentinelConfig = new RedisSentinelConfiguration(SENTINEL_MASTER, sentinelNodesSet);
      if (REDIS_SECURITY_ENABLED) {
          sentinelConfig.setPassword(REDIS_PASSWORD);
      }
      JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory(sentinelConfig);
      return jedisConnectionFactory;
    } catch (redis.clients.jedis.exceptions.JedisConnectionException re) {
      logger.error("Unable to initialize Redis connection factory", re);
      return null;
    }
}

On the other:

public RedisTemplate getRedisTemplate() {
  // We can assume that both methods are defined in the same class, 
  // although it is not necessary
  final RedisConnectionFactory redisConnectionFactory = this.jedisConnectionFactory();
  if (redisConnectionFactory == null) {
    return null;
  }

  final RedisTemplate redisTemplate = new StringRedisTemplate(redisConnectionFactory);
  return redisTemplate;
}

You can use this RedisTemplate in the way you consider appropriate and, of course, cache and reuse it as necessary.

These methods can be defined in a service or helper class created for this task and, of course, not in your configuration classes.

How to prevent fail fast on external connection failure like Redis server in Spring boot?

1 Answers