The driver does not deal with changing redis IP addresses in cluster mode

[akshaymankar]

Hedis loses connection to a redis cluster, if the redis cluster nodes change their IP addresses during a restart.

How to reproduce the problem

1. Start haskell app which talks to redis in cluster mode
2. Restart each redis node in a way that IP address of each node changes. This happens when using bitnami redis-cluster helm chart.
3. See that the app cannot talk to redis anymore

While the app cannot talk to redis, we get these kinds of error messages:

service: Network.Socket.connect: <socket: 51>: does not exist (No route to host)

Impact

This makes using hedis in Kubernetes very difficult.

Speculation

This could be a result of using socketToHandle here. The docs for the function say:

Caveat Handle is not recommended for network programming in Haskell, e.g. merely performing hClose on a TCP socket won't cooperate with peer's gracefulClose, i.e. proper shutdown sequence with appropriate handshakes specified by the protocol.

It could also be some other problem, this is just speculation.

Additional Info

I create ConnectInfo like this:

Redis.defaultConnectInfo
  { Redis.connectHost = "<kubernetes-service-name>"
    Redis.connectPort = 6379,
    Redis.connectTimeout = Just (secondsToNominalDiffTime 5),
    Redis.connectMaxConnections = 100
  }

And connect to redis like this:

Redis.connectCluster connInfo

This connection does not use TLS.

[k-bx]

Please see #184

[flokli]

@michaelglass, #168 suggests you're using cluster support quite extensively. Is this something you could take a look into?

[aravindgopall]

Need to run refreshShardMap which would get the new cluster info (by invoking cluster info call to redis). But in your case how will this query even run, you would need at least one node's static IP to run this query and get info which can be used to connect to all the nodes with new IPs.

PS: the current connections being held in pool might be still used and removed after only exceptions are thrown.

[michaelglass]

sorry about the late response. Maybe @stoeffel or @omnibs might have more context. I don't write much Haskell anymore 😔

[aravindgopall]

For anyone else looking here, we have fork in juspay (https://github.com/juspay/hedis) with some of these fixes done already. Would like to upstream them in sometime.

[ysangkok]

@aravindgopall Glad to see you forked. Is it the fix/connection branch people are supposed to use? Why is it 104 commits behind? Would you be interested in taking over hedis maintainership?

[aravindgopall]

Right now it is fix/connection branch, we are testing the changes in our setup. Once done will merge everything to cluster branch followed by master.

Why is it 104 commits behind?
The cluster branch have all of the changes before being merge to upstream hedis (done by @alexjg), there is a PR waiting to make it up to date with upstream. This all might take a week or two to settle.

@ysangkok Would be very much interested on this.

[aravindgopall]

JFI. fix/connection branch is up to date with upstream master and being tested. Once done will make merge to master and release.

[aravindgopall]

JFI, cluster branch have the latest code with the above fix and also some other fixes in Juspay Fork.

@ysangkok how do you want me to continue, shall i create a PR to upstream or we can continue on juspay fork.

[jbrechtel]

JFI, cluster branch have the latest code with the above fix and also some other fixes in Juspay Fork.

@ysangkok how do you want me to continue, shall i create a PR to upstream or we can continue on juspay fork.

I would suggest creating a PR to this repo and we can get it merged.

[jbrechtel]

Right now it is fix/connection branch, we are testing the changes in our setup. Once done will merge everything to cluster branch followed by master.

Why is it 104 commits behind?
The cluster branch have all of the changes before being merge to upstream hedis (done by @alexjg), there is a PR waiting to make it up to date with upstream. This all might take a week or two to settle.

@ysangkok Would be very much interested on this.

I mistakenly offered to maintain this library some months ago and it turns that I simply don't have the time. Could you email me and we can discuss handing over maintainership?

Hi @aravindgopall - I mistakenly offered to maintain this library some months ago and it turns that I simply don't have the time. Could you email me and we can discuss handing over maintainership? james at flipstone . com

[stefanwire]

Need to run refreshShardMap which would get the new cluster info (by invoking cluster info call to redis). But in your case how will this query even run, you would need at least one node's static IP to run this query and get info which can be used to connect to all the nodes with new IPs.

PS: the current connections being held in pool might be still used and removed after only exceptions are thrown.

This issue won't be solved by refreshing the shard map from IP addresses existing in the map: after Helm updated the Redis cluster, all Redis nodes will have new IP addresses assigned. That is, no IP address stored in the available shard map would connect to a then running Redis node. The only way to reconnect the client is by resolving a new valid IP address via the Redis cluster's DNS name and then obtain an entirely new shard map.

For our use-case, we retain the DNS name outside Hedis. This is suboptimal, since the DNS name is passed to Hedis for establishing the initial connection and could be retained for failovers when all IPs have become invalid. For instance, due to Helm being a villain.

[aravindgopall]

@stefanwire This is a valid scenario. Right now it resolves this issue where at least one of the Redis node persisted the ip address. Generally helpful in more frequent scenarios like failovers, scaling etc..

[stefanwire]

Feel free to check and adapt the code linked in my previous comment to solve the issue where all IPs wouldn't connect anymore. Also, check my open PR which adds the CLUSTER INFO command in order to improve robustness for Redis in cluster mode. We would be happy to use Hedis without local wrappers when this is done and merged.