[stable/redis] Making chart deployment HA with Sentinels?

[pentago]

Any chance the stable/redis chart can be refactored to provide genuine HA setup with Redis Sentinels (proper failover and slave promotion to master nodes)?

Official stable/redis-ha chart isn't working properly, it's not very production-friendly and PRs are merged at quite slow pace.

I think plenty of people would prefer Bitnami-built Redis HA chart with customizable persistence, unlike incubator/redis-cache chart which does Sentinels but not persistence too (this is a big deal for lots of deployments and companies). This chart could be used as a starting poing I guess.

Thanks in advance and looking forward to hearing opinions.

[javsalgar]

I have my concerns with this. Basically when the a slave gets promoted to master. How will the k8s services point to the correct master? At the moment, the solution that redis-ha proposes is to have some logic in the container that talks to the kubernetes API server and updates the labels accordingly. I am not very convinced by this approach as it is a bit obscure. That's the main reason we prefer using the native kubernetes failover solution in the redis chart.

@prydonius @sameersbn what's your take on this?

[pentago]

I just to be clear, I wasn't referring to "stable" redis-ha chart as a direction to go but incubator/redis-cache chart.

Also, do you @javsalgar think stable/redis failover is absolutely the best possible approach to the problem?

In fact, when I look into stable/redis chart it's totally unclear how it handles failover.
What happens exactly when and if the master fails. I understand that if pod is unhealthy it gets evicted from handling traffic from the service but do any of healthy slaves gets promoted to the master automatically? I guess not, at least it's not clear how and if at all occurs.

UPDATE:
Yeah, slaves are only read-only after master dies. That's not really HA, more like a poor failover solution, in a similar way as mysql/mariadb charts work. My incentive to start this issue is having Read/Write being available after master node dies.

The web deserves this! :)

[javsalgar]

Hi,

I did not know about redis-sentinel-micro. I suppose that it would be responsible of updating the k8s labels for the services to point to the new master. If it's properly maintained, using it could make sense to have master promotion. Thanks for sharing :)

[pentago]

It doesn't need this be micro per se, it can be literally anything thats puvlic and vetted as far as I'm concerned. Any other thoughts?

[javsalgar]

Ping @prydonius @sameersbn

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[javsalgar]

PR helm/charts#13112 should cover your use case

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[javsalgar]

@pentago could you confirm that helm/charts#13112 is what you were looking for?

[pentago]

Looks like it but since asking this I used official redis chart which has failover option. Not sure how these two compare but will test when I get some time. Thanks!

[javsalgar]

Let us know any feedback you have :)

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.