etcd defrag + backup: Avoid too many leader changes

[garloff]

As k8s cluster user, I want the k8s control plane to always be responsive, stable and safe.

We have a nightly job to defragment etcd and back it up on all control plane nodes, randomized a bit, so the defragmentation does not happen all at the same time.
This has been in existence for many months already, but due to a missing --now in systemctl enable, it has not really been active before.
As @matofeder points out, the defragmentation may block access to etcd for a while (seconds on typically sized etcd DBs), causing etcd leader changes (on multi-node etcd clusters) or temporary kube-api failures (on single-node etcd clusters).

Things to consider:

1. Possibly stronger protection from concurrent defragmentation on multiple etcd nodes by scheduling all etcd nodes from the leader instead of relying on the configured randomness in the timer start time.
2. Scheduling the leader etcd defragmentation last, as it will likely cause a leader change and we want to minimize these. (Starting with the leader would cause several leader changes ...)
3. Skipping the leader's defragmentation (for up to a week or infinitely?) to cause less leader changes?
4. Skipping defragmentation on single-node etcd installations?
5. Leaving this disabled for R4 and do more real-world tests before R5. (This is not without risk either, we have seen heavily fragmented etcds causing trouble in real-life already.)

[garloff]

To point 1: It should be noted that the backup and fstrim operations can not be done centrally but would need a local job on the non-leader control-plane nodes, possibly triggered via ssh.

[garloff]

Point 2 is easy to do.

[garloff]

Points 3+4: If we decide to simply skip defrag on the etcd leader (infinitely), we'd cover both of these points with a single change.

[garloff]

Your feedback is welcome, especially @matofeder, @chess-knight, @batistein, @janiskemper, @ajfriesen, @flyersa, @curx, @fynluk, @mxmxchere.

[batistein]

@guettli this is what we talked about. maybe we could contribute here

[guettli]

I think the best solution would be an official solution from etcd.io.

Maybe it is enough to update the docs: I created an issue for that: etcd-io/etcd#15477

Maybe it is enough to skip defragmentation if the current instance is the leader. After N hours the cron-job would try it again. AFAIK defragmentation is not that pressing, so that this simple solution might be ok.

There is a K8s cronjob etcd-defrag-cronjob, but this depends on Prometheus. I personally would prefer a solution which does not depend on a third party tool like prometheus.

[matofeder]

There is a K8s cronjob etcd-defrag-cronjob, but this depends on Prometheus. I personally would prefer a solution which does not depend on a third party tool like prometheus.

I agree. But the script from the etcd-defrag-cronjob project could be adopted as a solid base for our implementation.

I would propose the following (with the respect to points 1..3 in the description):

Write a short script that will be deployed to each control plane node and executed periodically (at the same time), e.g. the etcd-defrag.service could be adjusted here and instead of etcdctl ... defrag, e.g. the defrag.sh script will be executed periodically. The script does the following:

1. Check if the control plane node where the script is executed is an ETCD leader node
2. Finish if the node is not ETCD leader
3. Executes the following if the node is ETCD leader:
   • Defrag all non-leader ETCD nodes, step by step
   • Change the leader node with etcdctl move-leader to the defragmentation completed ETCD node (optional, but we don't have to wait election-timeout, which could eliminate the loss of writes already sent to the old leader , see docs)
   • Defrag the node where the script is executed (currently not a leader)

Next periodic execution does the same.

4. Skipping defragmentation on single-node etcd installations?

I would say YES as this may cause some unwanted issues on the client side. Optionally, a "special" flag could be introduced to allow this, e.g. --allow-etcd-single-node-defrag

[garloff]

I like the described approach by @matofeder.

I have not yet looked at the implementation of defrag.sh and how it compares to what we are currently doing.

In addition to defragmenting, we do create a local backup (for manual disaster recovery) and do a discard (fstrim) on the filesystem to combat FTL fragmentation. I think both are useful. Both are also fine to only happen on the previous leader as we will change the leader on a daily basis at least (in absence of other leader-changing events). We would need to ensure that the leader changes don't ping-pong between just two etcd nodes but are either arbitrary or round-robin.

[garloff]

PS: We could just document that single control-plane nodes have the risk of slowly degrading due to fragmentation and that they are not meant for long-term operation. Work-arounds would be manual intervention (not recommended) or periodic temporary upgrades to a three control-plane-node scenario over night.

[garloff]

As we have the etcd fragmentation unconditionally enabled since merging #355 a few days ago, I want to ensure we don't cause trouble for users of the main branch and introduce a quick fix.
PR #387 does simply exit the defrag script on the leader, so we only defragment (and backup) etcd on the non-leader etcd cluster members. If we happen to get occasional leader changes, this would even almost create the desired state.
This should be significantly safer than the current state in main, but of course not as good as the approach depicted above. We should merge it IMVHO, if the better solution can not be created, validated and merged within the next few days.
@matofeder, @chess-knight, @batistein, @guettli -- thanks for your great input!
Any of you looking at implementing the suggested approach in the next days?

[matofeder]

PS: We could just document that single control-plane nodes have the risk of slowly degrading due to fragmentation and that they are not meant for long-term operation. Work-arounds would be manual intervention (not recommended) or periodic temporary upgrades to a three control-plane-node scenario over night.

I agree

We should merge it IMVHO, if the better solution can not be created, validated and merged within the next few days.

Let's merge #387 as a hotfix.

Any of you looking at implementing the suggested approach in the next days?

I will take care of that.

[guettli]

BTW, Gardener has related repository, and they plan this feature: Defragmentor of backup-restore should also consider the etcd db size along with scheduled defrag