Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required on CLIENT and READONLY command #2996

Closed
question0914 opened this issue Sep 28, 2024 · 11 comments · Fixed by #3035
Closed

io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required on CLIENT and READONLY command #2996

question0914 opened this issue Sep 28, 2024 · 11 comments · Fixed by #3035
Labels
type: bug A general bug
Milestone
6.5.0.RELEASE

Comments

@question0914
Copy link

question0914 commented Sep 28, 2024 •
edited
Loading

Bug Report

Current Behavior

When attempting to connect to an AWS MemoryDB cluster using Lettuce version 6.4.0.RELEASE, an error occurs related to authentication for the CLIENT and READONLY commands. This issue persists regardless of the protocol version used (RESP2 or RESP3). However, requests to the MemoryDB cluster still succeed and return responses despite this error.

2024-09-28 01:00:32.236 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0xbefb39e8, /100.69.22.144:35512 -> memorydb-finance-0011-001.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.135:6379, epid=0x75c, chid=0x761] Completing command AsyncCommand [type=CLIENT, output=StatusOutput [output=null, error='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]

2024-09-28 02:07:12.067 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-9] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0xb411c837, /100.69.84.97:51846 -> memorydb-finance-0003-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.232.43:6379, epid=0x37b, chid=0x183d9] Completing command AsyncCommand [type=READONLY, output=StatusOutput [output=null, error='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]
Stack trace 
// your stack trace here;
// Following stack trace occurred once the client connected to the cluster
2024-09-28 00:44:09.819 yrid= x-request-id= uri=/v8/finance/spark?symbols=VZ method=GET [qtp1032000752-103] DEBUG io.lettuce.core.AbstractRedisClient - Connecting to Redis at clustercfg
.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com:6379
2024-09-28 00:44:09.922 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, [id: 0xed1622ae] (inactive), e
pid=0x1, chid=0x1] channelRegistered()
2024-09-28 00:44:09.960 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] write(ctx, [AsyncCommand [type=CLIENT, output=StatusOutput [output=null, error='null
'], commandType=io.lettuce.core.protocol.Command], AsyncCommand [type=CLIENT, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]], promise)
2024-09-28 00:44:09.962 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandEncoder - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379] writing command AsyncCommand [type=CLIENT, output=StatusOutput [output=null, error='null'], commandType=
io.lettuce.core.protocol.Command]
2024-09-28 00:44:09.962 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandEncoder - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379] writing command AsyncCommand [type=CLIENT, output=StatusOutput [output=null, error='null'], commandType=
io.lettuce.core.protocol.Command]
2024-09-28 00:44:09.965 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] write(ctx, AsyncCommand [type=HELLO, output=GenericMapOutput [output=null, error='nu
ll'], commandType=io.lettuce.core.protocol.Command], promise)
2024-09-28 00:44:09.965 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandEncoder - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379] writing command AsyncCommand [type=HELLO, output=GenericMapOutput [output=null, error='null'], commandTy
pe=io.lettuce.core.protocol.Command]
2024-09-28 00:44:09.982 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] Received: 214 bytes, 3 commands in the stack
2024-09-28 00:44:09.983 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] Stack contains: 3 commands
2024-09-28 00:44:09.984 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.RedisStateMachine - Decode done, empty stack: true
2024-09-28 00:44:09.984 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> clustercfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] Completing command AsyncCommand [type=CLIENT, output=StatusOutput [output=null, error='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]
024-09-28 00:44:09.986 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] Stack contains: 2 commands
2024-09-28 00:44:09.986 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.RedisStateMachine - Decode done, empty stack: true
2024-09-28 00:44:09.986 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> cluster
cfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] Completing command AsyncCommand [type=CLIENT, output=StatusOutput [output=null, erro
r='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]
2024-09-28 00:44:09.988 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.RedisHandshake - Error applying connection metadata
java.util.concurrent.CompletionException: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
        at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1298)
        at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
        at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1034)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
        at io.lettuce.core.protocol.AsyncCommand.doCompleteExceptionally(AsyncCommand.java:143)
        at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:124)
        at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:115)
        at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:749)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:684)
        at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:601)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
           at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)
Caused by: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.
        at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:151)
        at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:120)
        ... 28 common frames omitted
2024-09-28 00:44:09.988 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-1] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x47827c96, /100.69.84.97:51222 -> clustercfg.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.226.99:6379, epid=0x1, chid=0x1] Stack contains: 1 commands


// Following occurred after app deployed for about an hour
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.ConnectionWatchdog - [channel=0xc315e35b, /100.69.84.97:54240 -> mem
orydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, last known addr=memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.
amazonaws.com/10.157.193.94:6379] scheduleReconnect()
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.ConnectionWatchdog - [channel=0xc315e35b, /100.69.84.97:54240 -> mem
orydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, last known addr=memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.
amazonaws.com/10.157.193.94:6379] Reconnect attempt 318, delay 2991ms
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0xbec8b645, /100.69.84.97:53084 -> memoryd
b-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, epid=0x29b, chid=0x183bf] channelInactive()
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.DefaultEndpoint - [channel=0xc315e35b, /100.69.84.97:54240 -> memory
db-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, epid=0x29b] deactivating endpoint handler
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.DefaultEndpoint - [channel=0xc315e35b, /100.69.84.97:54240 -> memory
db-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, epid=0x29b] notifyQueuedCommands adding 0 command(s) to buffer
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0xbec8b645, /100.69.84.97:53084 -> memoryd
b-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, epid=0x29b, chid=0x183bf] channelInactive() done
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.ConnectionWatchdog - [channel=0xc315e35b, /100.69.84.97:54240 -> mem
orydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, last known addr=memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.
amazonaws.com/10.157.193.94:6379] channelInactive()
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.ConnectionWatchdog - [channel=0xc315e35b, /100.69.84.97:54240 -> memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, last known addr=memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379] scheduleReconnect()
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.ConnectionWatchdog - [channel=0xc315e35b, /100.69.84.97:54240 -> memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, last known addr=memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379] Skipping scheduleReconnect() because I have an active channel
2024-09-28 02:07:11.668 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-6] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0xbec8b645, /100.69.84.97:53084 -> memorydb-finance-0006-003.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.193.94:6379, epid=0x29b, chid=0x183bf] channelUnregistered()
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memorydb-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Received: 249 bytes, 4 commands in the stack
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memoryd
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memoryd
b-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Completing command AsyncCommand [type=READONLY, output=StatusOu
tput [output=null, error='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memoryd
b-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Stack contains: 3 commands
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.RedisStateMachine - Decode done, empty stack: true
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memoryd
b-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Completing command AsyncCommand [type=CLIENT, output=StatusOutp
ut [output=null, error='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memoryd
b-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Stack contains: 2 commands
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.RedisStateMachine - Decode done, empty stack: true
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memoryd
b-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Completing command AsyncCommand [type=CLIENT, output=StatusOutp
ut [output=null, error='NOAUTH Authentication required.'], commandType=io.lettuce.core.protocol.Command]
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.RedisHandshake - Error applying connection metadata
java.util.concurrent.CompletionException: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
        at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1298)
        at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
        at java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1034)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
        at io.lettuce.core.protocol.AsyncCommand.doCompleteExceptionally(AsyncCommand.java:143)
        at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:124)
        at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:115)
        at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:749)
        at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:684)
        at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:601)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)
Caused by: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.
        at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:151)
        at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:120)
        ... 28 common frames omitted
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memorydb-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Stack contains: 1 commands
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.RedisStateMachine - Decode done, empty stack: true
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] DEBUG io.lettuce.core.protocol.CommandHandler - [channel=0x9fe575db, /100.69.84.97:56996 -> memorydb-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com/10.157.198.169:6379, epid=0x33, chid=0x183c1] Completing command AsyncCommand [type=HELLO, output=GenericMapOutput [output={server=redis, version=7.1.1, proto=3, id=2719043, mode=cluster, role=replica}, error='null'], commandType=io.lettuce.core.protocol.Command]
2024-09-28 02:07:11.669 yrid= x-request-id= uri= method= [lettuce-epollEventLoop-4-4] WARN  io.lettuce.core.protocol.ConnectionWatchdog - Cannot reconnect to [memorydb-finance-0010-002.memorydb-finance.zek3dz.memorydb.us-east-1.amazonaws.com:6379]: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.
java.util.concurrent.CompletionException: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)

Input Code

Input Code 
// your code here;
// How we initialize the client
@Override
    public RedisClusterClient get() {
        LOGGER.info("Initializing RedisClusterClient");
        RedisURI redisURI = RedisURI.Builder.redis(hostName)
                .withPort(6379)
                .withAuthentication(redisCredentialProvider)
                .withSsl(true)
                .build();
        
        ClientResources resources = DefaultClientResources.builder()
                .reconnectDelay(Delay.fullJitter(
                        Duration.ofMillis(100),     // minimum 100 millisecond delay
                        Duration.ofSeconds(5),      // maximum 5 second delay
                        100, TimeUnit.MILLISECONDS) // 100 millisecond base
                ).build();

        RedisClusterClient client = RedisClusterClient.create(resources, redisURI);
        client.setOptions(
                ClusterClientOptions.builder()
                        .autoReconnect(true)
                        .socketOptions(SocketOptions.builder()
                                .connectTimeout(Duration.ofMillis(connectTimeoutMillis))
                                .keepAlive(true)
                                .build()
                        )
                        .timeoutOptions(TimeoutOptions.builder()
                                .fixedTimeout(Duration.ofMillis(commandTimeoutMillis))
                                .build()
                        )
                        .topologyRefreshOptions(ClusterTopologyRefreshOptions.builder()
                                .enablePeriodicRefresh(Duration.ofSeconds(10))
                                .enableAllAdaptiveRefreshTriggers()
                                .dynamicRefreshSources(true)
                                .build()
                        )
                        // https://redis.io/docs/latest/commands/cluster-nodes/
                        .nodeFilter(it ->
                                !(it.is(RedisClusterNode.NodeFlag.FAIL)
                                        || it.is(RedisClusterNode.NodeFlag.EVENTUAL_FAIL)
                                        || it.is(RedisClusterNode.NodeFlag.HANDSHAKE)
                                        || it.is(RedisClusterNode.NodeFlag.NOADDR)
                                        || (it.is(RedisClusterNode.NodeFlag.REPLICA) && it.getReplOffset() == -1L)))
                        .build());

        Runtime.getRuntime().addShutdownHook(new Thread(client::shutdown));
        return client;
    }


// How we fetch the credentials in RedisStaticCredentialsProvider 
  
 private volatile RedisCredentials cachedCredentials = null;


 @Override
    public Mono<RedisCredentials> resolveCredentials() {
        // Check if credentials are already fetched
        if (cachedCredentials != null) {
            return Mono.just(cachedCredentials);
        }
        fetchCredentials();
        return Mono.just(cachedCredentials);
    }

 private void fetchCredentials() {
        LOGGER.info("Resolving Credentials");
        // Check if credentials are already fetched
        if (cachedCredentials != null) {
            return;
        }
        synchronized (this) {
            if (cachedCredentials == null) { // Double-checked locking
                if (this.yKeyKeyClient != null) {
                    // ...
                    if (usernameSecret != null && passwordSecret != null) {
                        LOGGER.info("Get credentials successfully, duration={}", System.currentTimeMillis() - startTime);
                        cachedCredentials = RedisCredentials.just(usernameSecret, passwordSecret);
                    } else {
                        LOGGER.error("Null username or password secrets returned");
                    }
                } else {
                    LOGGER.error("YKeyKeyClient is not successfully initialized");
                }
            }
        }
    }

Expected behavior/code

The CLIENT and READONLY commands should execute successfully without authentication errors.

Environment

  • Lettuce version(s): [6.4.0.RELEASE]
  • Redis version: [7.1]

Possible Solution

Additional context
@question0914 question0914 changed the title io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required on CLIENT command io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required on CLIENT and READONLY command Sep 28, 2024
@tishun tishun added for: team-attention An issue we need to discuss as a team to make progress status: waiting-for-triage labels Oct 8, 2024
@tishun tishun added the type: bug A general bug label Oct 22, 2024
@tishun
Copy link
Collaborator

tishun commented Oct 22, 2024

Could be related to #2823

@WenyuZhang1992
Copy link

Hello @tishun , is there any workaround for this? We have upgraded to version 6.4.0.RELEASE

@tishun
Copy link
Collaborator

tishun commented Oct 29, 2024

Just a quick test, because we are still unable to reproduce the issue, could you attempt the solution provided in #2817 (comment) ?

@WenyuZhang1992
Copy link

The issue happens after we upgrade to version 6.4.0.RELEASE, does this workaround mentioned still required?

@WenyuZhang1992
Copy link

WenyuZhang1992 commented Oct 30, 2024 •
edited
Loading

This old issue looks kinda relevant: #1201 but looks the config pingBeforeActivateConnection is default to true now

@tishun
Copy link
Collaborator

tishun commented Oct 30, 2024

Apologies, did not give the full explanation and I see my comments were confusing.

The log you sent clearly shows that there is a bug in the order of the commands that are sent during handshake. Typically the driver would always first send an HELLO command (and authenticate) and then call the CLIENT commands to bootstrap the connection with some metadata (about the driver name and version, etc.). in your case the CLIENT is sent first, before the authentication happens.

Now this metadata is sent in a fire-and-forget way since 6.4.0.RELEASE so you would only be seeing these issues in the log, but the application should not fail (this is done because older versions of the Redis server did not support them and this information is not necessary for the driver to function properly).

If you apply the suggested code you would disable this metadata being sent and it should stop the errors from showing up in the logs until we manage to reproduce the issue locally and fix it. We managed to achieve it once, but not since, so it might take some time for us to find the issue.

Please let me know if:

  • you have issues using the driver after the message is shown in the log (you should not)
  • you disabled reporting the driver version and library and are still getting the issue shown (you should not)

Just realized that you should also disable reporting the client name (not only the library name and version). So the complete (temporary) fix should look like:

RedisURI redisUri = RedisURI.Builder.redis("my.redis.instance.com", 6379)
    .withLibraryName("")
    .withLibraryVersion("")
    .withClientName("")
    .build();
RedisClient redisClient = RedisClient.create(redisUri);

@WenyuZhang1992
Copy link

Thanks @tishun for the detailed explanations, it does make sense to me now for CLIENT command's error. But still not sure what should we do regarding the NOAUTH logging for READONLY command? If you can provide a workaround for READONLY will be great!

We do notice an issue(even though not 100% sure if it's related): we currently set REPLICA_PREFERRED and expect most read-only commands can be processed by replica nodes. But we realized especially during read traffic spike, master node starts to handle much more read-only commands and causing CPU bottleneck.

@ggivo ggivo linked a pull request Oct 30, 2024 that will close this issue
io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required on CLIENT and READONLY command #3035
Merged
4 tasks
@ggivo ggivo mentioned this issue Oct 31, 2024
Merged
4 tasks
@ggivo
Copy link
Contributor

ggivo commented Oct 31, 2024 •
edited
Loading

Hi @WenyuZhang1992

Actually it make sense also for the READONLY command. Unfortunately it can not be toggled off similar to CLIENT since READONLY is not kind of optional one. It is again send during applyPostHandshake, and if you check the logs you will see that both CLIENT and READONLY commands are executed before the HELLO one.

If the READONLY fails it will not mark the channel as readonly and this can relate to second issue mentioned.

We have drafted a fix that aims to fix the order of command execution e.g. HELLO -> READONLY -> CLIENT (optional)
Could you try it out and confirm if it resolves the issue in your case it will be helpfull

Hope it helps

@tishun tishun removed for: team-attention An issue we need to discuss as a team to make progress status: waiting-for-triage labels Oct 31, 2024
@tishun tishun added this to the 6.5.0.RELEASE milestone Oct 31, 2024
@tishun
Copy link
Collaborator

tishun commented Oct 31, 2024

I will also port this for 6.4.1.RELEASE

@WenyuZhang1992
Copy link

Thanks @tishun @ggivo , I can just upgrade to 6.5.0.RELEASE to try this out?

@tishun
Copy link
Collaborator

tishun commented Oct 31, 2024

Thanks @tishun @ggivo , I can just upgrade to 6.5.0.RELEASE to try this out?

Yes, if upgrading works for you then this should do the trick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type: bug A general bug
Projects
None yet
Milestone
6.5.0.RELEASE
4 participants
@tishun @question0914 @WenyuZhang1992 @ggivo