Rueidis Compat Client vs Go-Redis latency

[t-kruth]

Currently we're looking to leverage rueidis in place of the go-redis package. We leveraged it in a service and noticed that the roundtrip latency spiked along with our processing of events.

Currently running version 7.1 redis in AWS Elasticache in clustermode

we're using mostly default config options with the exception of
BlockingPoolSize = 100
Dialer.Timeout = 1s
SenToReplicas = cmd.IsReadOnly()
and leveraging TLS

the client is created,
we add a hook for timing
and then leverage the rueidiscompt.NewAdapter so we can leverage the already existing code for redis operations

[rueian]

we add a hook for timing

Do you mean rueidishook?

so we can leverage the already existing code for redis operations

What redis operations are you using?

We leveraged it in a service and noticed that the roundtrip latency spiked along with our processing of events.

Would it be possible to add more CPU cores to your application and see if those spikes go down?

[Ghilteras]

@rueian does the PipelineMultiplex need to be filled before it sends it to the cluster? We have 90 nodes, so if we set it to 2 that's 360 connections per pod. Does it mean the pod needs to fill the pipe before it gets to Redis? Our p90, p95 and p98 are fine compared to go-redis, but our p99 is way above go-redis. We see spikes of more than 2 SECONDS when we switch to Rueidis (the target is same region same VPC cluster) and it looks like there's nothing else we can tune about the client. We really want to migrate to Rueidis, but unless this is a known problem with a known solution we'll have to stay with go-redis because it just seems faster (contrary to what you showed in your benchmarks). It's not possible to keep running Spikes about this migration forever just to find out why our p99 is so slow compared to go-redis. Unless you can think of something glaring we missed, we'll have to give up..

[rueian]

Hi @Ghilteras, PipelineMultiplex is an argument that decides how many connections we use for pipelining commands to one Redis instance. For example, if PipelineMultiplex = 2, then we prepare 4 connections for pipelining. It is like a connection pool but connections in that pool are shared.

So, the p99 latency issue usually comes from the nature of pipeline queuing, aka the head-of-line blocking, caused by large data transfers in or out of the connection. I hope I have a smarter way to handle this issue, but currently, the better way might be just disabling the auto pipelining. I will add an option DisableAutoPipelining to the next version.

[Ghilteras]

@rueian I guess my question is why this is not visible in your benchmarks? I mean we can keep the pipeline feature and eat the p99 lag knowing that it would benefit us for the very big waves of events if we can disable it and prove that the p99 latency go away.

[rueian]

I think it's mostly because the benchmark didn't involve large data transfer.

Also, the numbers in the benchmark were ns/op which was throughput not latencies.

This one #93 was tested on two GCP n2-highcpu-2 machines with PipelineMultiplex = 2. It showed better latencies when concurrency >= 8. But didn't have large data transfers as well.

[rueian]

Hi @Ghilteras, we now have the DisableAutoPipelining option on v1.0.48-alpha.1. The benchmark shows slightly better latencies and throughput on two GCP n2-highcpu-2 across concurrency 2, 4, 8, 16, and 32. CPU usage is also slightly lower. Please let me know if this helps.