Error:: redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster

[nitinware]

Expected behavior

I am expecting onc we establish the connection with AWS ElasticCache Redis cluster successfully we should not see any of these connection errors and we should not have to restart our application/service.

Actual behavior

• We recently moved from local caching to AWS Elasticache Redis Caching where our application/service a SpringBoot application connects to Redis for read/write operations with read operations being the majority of traffic to Redis.
• We are seeing this error across multiple env's where we have deployed our application and redis cluster.
• Currently we are addressing this error by bouncing/restarting out application/service, post restart we don't see this error.

Exception: No reachable node in cluster; nested exception is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
org.springframework.data.redis.RedisConnectionFailureException: No reachable node in cluster; nested exception is redis.clients.jedis.exceptions.JedisNoReachableClusterNodeException: No reachable node in cluster
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:65)
	at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:42)
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44)
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42)
	at org.springframework.data.redis.connection.jedis.JedisClusterConnection.convertJedisAccessException(JedisClusterConnection.java:787)
	at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.convertJedisAccessException(JedisClusterStringCommands.java:528)
	at org.springframework.data.redis.connection.jedis.JedisClusterStringCommands.get(JedisClusterStringCommands.java:67)
	at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:267)
	at org.springframework.data.redis.cache.DefaultRedisCacheWriter.lambda$get$1(DefaultRedisCacheWriter.java:126)
	at org.springframework.data.redis.cache.DefaultRedisCacheWriter.execute(DefaultRedisCacheWriter.java:304)
	at org.springframework.data.redis.cache.DefaultRedisCacheWriter.get(DefaultRedisCacheWriter.java:126)
	at org.springframework.data.redis.cache.RedisCache.lookup(RedisCache.java:89)
	at org.springframework.cache.support.AbstractValueAdaptingCache.get(AbstractValueAdaptingCache.java:58)
	at org.springframework.cache.transaction.TransactionAwareCacheDecorator.get(TransactionAwareCacheDecorator.java:80)
	at org.springframework.cache.interceptor.AbstractCacheInvoker.doGet(AbstractCacheInvoker.java:73)
	at org.springframework.cache.interceptor.CacheAspectSupport.findInCaches(CacheAspectSupport.java:571)
	at org.springframework.cache.interceptor.CacheAspectSupport.findCachedItem(CacheAspectSupport.java:536)
	at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:402)

Steps to reproduce:

Redis / Jedis Configuration

Jedis version:

3.6.3

Redis version:

5.0.6

Java version:

openjdk version "11.0.8" 2020-07-14 LTS
OpenJDK Runtime Environment Zulu11.41+24-SA (build 11.0.8+10-LTS)
OpenJDK 64-Bit Server VM Zulu11.41+24-SA (build 11.0.8+10-LTS, mixed mode)
openjdk_11.0.8_11.41.24_x64

[sazzad16]

Try Jedis version 3.10.0.

[nitinware]

thnx for your reply, I will try with jedis version 3.10.0 and update here with the results.

[yufeldman]

@sazzad16 - regarding trying version 3.10.0 - are there any related fixes. Just trying to see the rationale for trying that version

[sazzad16]

@yufeldman

[Shuang-Wang-1118]

Hi @sazzad16 , we faced the same issue and after we put above update, the issue is gone, but from RCA perspective, do you have insights of why the 3.10 will fix our No reachable node in cluster side, which exact code fix this issue?

[sazzad16]

@Shuang-Wang-1118 You can check #3370 (84b485d) for the exact code. This is actually the backported version of #2643 which you can check for more information.