Fix #9696 - apiserver outage when replacing or scaling control plane nodes

[holmesb]

Fixed by bouncing apiserver static pods sequentially instead of all at once when there are etcd node changes. Retain the faster, old method for use by non-HA apiserver.

No longer change running apiserver static pods (and restart them) if only the order of etcd servers have changed.

/kind bug

What this PR does / why we need it:
Fixes loss of apiserver when are etcd nodes are scaled, or CP1 is replaced.

Which issue(s) this PR fixes:
Fixes #9696

Does this PR introduce a user-facing change?:
NONE

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: holmesb (646bc92)

[k8s-ci-robot]

Hi @holmesb. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

Keywords which can automatically close issues and at(@) or hashtag(#) mentions are not allowed in commit messages.

The list of commits with invalid commit messages:

• 0a9b600 Fix apiserver outage when replacing or scaling control plane nodes #9696 by bouncing apiserver static pods sequentially instead of all at once when there are etcd node changes. Retain the faster, old method for use by non-HA apiserver.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[holmesb]

Fixed author\CLA.

[holmesb]

CI has a "command-instead-of-module" Ansible lint error because I'm using sed & curl instead of lineinfile & uri. But we must query the local endpoint (127.0.0.1/healthz) before moving onto the next CP node (apiserver static pod). Can't use throttle at the block level, otherwise I'd couple a lineinfile with a uri task. Open to suggestions. Maybe I should split this into its own play, then I could use serial. Or can we just ignore this lint error?

[floryut]

CI has a "command-instead-of-module" Ansible lint error because I'm using sed & curl instead of lineinfile & uri. But we must query the local endpoint (127.0.0.1/healthz) before moving onto the next CP node (apiserver static pod). Can't use throttle at the block level, otherwise I'd couple a shell with a uri task. Open to suggestions. Maybe I should split this into its own play, then I could use serial. Or can we just ignore this lint error?

ignoring the error is fine for this case I'd say, as long as there is a valid reason it's fine for me

[holmesb]

Any news with this one @cyclinder @jayonlau? Be good to avoid downtime everytime control plane nodes are changed. Are no breaking changes.

[holmesb]

Any news with this pls @floryut ? K8s API server going offline every CP change isn't exactly "Production Ready". I'd like to avoid having to add this as a patch to Kubespray every release\build.

[floryut]

Any news with this pls @floryut ? K8s API server going offline every CP change isn't exactly "Production Ready". I'd like to avoid having to add this as a patch to Kubespray every release\build.

@holmesb Sorry I was pretty much waiting for you to fix the CI not passing 😆
Can you take a look ?

[holmesb]

The ansible lint message? I thought you said "ignoring the error is fine for this case I'd say". We're already using sed & curl elsewhere in this repo, so I can't envisage any issues.

[floryut]

The ansible lint message? I thought you said "ignoring the error is fine for this case I'd say". We're already using sed & curl elsewhere in this repo, so I can't envisage any issues.

Yes, ignoring the error, meaning adding a noqa in the code on this specific line 😄

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: holmesb (6d1a5c1)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: holmesb
Once this PR has been reviewed and has the lgtm label, please assign luckysb for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: holmesb (bad2f15)

[holmesb]

Passing CI now @floryut

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jcpunk]

Can you rebase off head?

[k8s-ci-robot]

Keywords which can automatically close issues and at(@) mentions are not allowed in the title of a Pull Request.

You can edit the title by writing /retitle in a comment.

When GitHub merges a Pull Request, the title is included in the merge commit. To avoid invalid keywords in the merge commit, please edit the title of the PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[holmesb]

Can you rebase off head?

Done @jcpunk

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[VannTen]

IMO, we should just revert #8253 instead. It's documented here that scale.yml it not intended for control-plane components, and you should use cluster/upgrade-cluster.
Reverting that PR would mean the api-server conf change is handled during the normal playbook upgrade path one at a time, and would avoid the mentioned down-time.

[VannTen]

I think this has been fixed by the linked PR above, so we'll close that.
Feel free to reopen if still needed.
Thanks for the work regardless o/
/close

[k8s-ci-robot]

@VannTen: Closed this PR.

In response to this:

I think this has been fixed by the linked PR above, so we'll close that.
Feel free to reopen if still needed.
Thanks for the work regardless o/
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.