Redis Timeout Exception - .NET Framework 4.8, everything async

[TimSalisbury]

Hi,

Every now and again we are receiving the following exception from StackExchange.Redis:

Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5703ms elapsed, timeout is 5000ms), command=SETEX, next: SET xxxxx, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 34, last-in: 0, cur-in: 0, sync-ops: 0, async-ops: 440495, serverEndpoint: xxxxx, conn-sec: 76835.98, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxxx(SE.Redis-v2.8.16.12844), PerfCounterHelperkeyHashSlot: 141, IOCP: (Busy=2,Free=998,Min=2,Max=1000), WORKER: (Busy=6,Free=32761,Min=2,Max=32767), v: 2.8.16.12844 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

This is while running on a .NET Framework 4.8 application, with every call being async.

Any ideas? I appreciate any help with this.

[philon-msft]

It's normal to receive occasional timeouts due to things like Redis server maintenance. We recommend implementing a command retry strategy using something like Polly

[TimSalisbury]

Thank you for the response!

I can add some more context:

This is running against an AWS ElastiCache Redis cluster with two nodes. From what I can tell, there was no maintenance during this period when we received this error. Along with that, there were several servers going to our Redis cluster during this time, and only one of the servers ended up with this error

[avmgan]

We do see similar issue retry already implemented.

2024-09-17 9:47:50 ERROR [Sdl.Web.Delivery.Caching.CacheHandlers.RedisCache.RedisCacheHandler1.Get] Redis Timeout 'xxxxxxx.use1.cache.amazonaws.com:6379' due to System.TimeoutException: Timeout performing EVAL, inst: 56, queue: 31, qu: 0, qs: 31, qc: 0, wr: 0, wq: 0, in: 131072, ar: 0, clientName: WEBCDPROD10C, IOCP: (Busy=1,Free=999,Min=8,Max=1000), WORKER: (Busy=106,Free=32661,Min=8,Max=32767), Local-CPU: unavailable (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) at Microsoft.Extensions.Caching.Redis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) at Microsoft.Extensions.Caching.Redis.RedisCache.GetAndRefresh(String key, Boolean getData) at Sdl.Web.Delivery.Caching.CacheHandlers.RedisCache.RedisCacheHandler1.<>c__DisplayClass13_0.b__0()
at Sdl.Web.Delivery.Caching.CacheHandlers.RedisCache.RedisCacheHandler1.RetryBlock[TResult](Func1 block, TResult& returnedValue).

2024-09-17 9:45:54 ERROR [Sdl.Web.Delivery.Caching.CacheHandlers.RedisCache.RedisCacheHandler1.Get] Redis Timeout xxxxx.cache.amazonaws.com:6379' due to System.TimeoutException: Timeout performing EVAL, inst: 191, queue: 62, qu: 0, qs: 62, qc: 0, wr: 0, wq: 0, in: 131072, ar: 0, clientName: WEBCDPROD10B, IOCP: (Busy=0,Free=1000,Min=8,Max=1000), WORKER: (Busy=356,Free=32411,Min=8,Max=32767), Local-CPU: unavailable (Please take a look at this article for some common client-side issues that can cause timeouts: https://github.com/StackExchange/StackExchange.Redis/tree/master/Docs/Timeouts.md) at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor1 processor, ServerEndPoint server) at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) at Microsoft.Extensions.Caching.Redis.RedisExtensions.HashMemberGet(IDatabase cache, String key, String[] members) at Microsoft.Extensions.Caching.Redis.RedisCache.GetAndRefresh(String key, Boolean getData) at Sdl.Web.Delivery.Caching.CacheHandlers.RedisCache.RedisCacheHandler1.<>c__DisplayClass13_0.b__0()
at Sdl.Web.Delivery.Caching.CacheHandlers.RedisCache.RedisCacheHandler1.RetryBlock[TResult](Func1 block, TResult& returnedValue).

Any ideas? I appreciate any help with this.

[isacruzramos]

hi @philon-msft, I have also experienced a couple of random time outs even when the Redis server load is low
we are using version 2.7.33 of the C# SDK, do we know how often can we hit this? Can correlate it to an specific maintenance window of the cache?

-------Exception information------
System.TimeoutException: Task did not complete within the TimeoutMs=5000
at Microsoft.Kailani.Common.TaskExtensions.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Kailani.Common.RedisCacheUtil.d__10.MoveNext()