Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Are kops clusters subject to Kubernetes known issue: etcd client balancer with secure endpoints #7816

Closed
Nuru opened this issue Oct 21, 2019 · 5 comments
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@Nuru
Copy link

Nuru commented Oct 21, 2019

Kubernetes documentation lists a "known issue": etcd client balancer with secure endpoints

The etcd v3 client, released in etcd v3.3.13 or earlier, has a critical bug which affects the kube-apiserver and HA deployments. The etcd client balancer failover does not properly work against secure endpoints. As a result, etcd servers may fail or disconnect briefly from the kube-apiserver. This affects kube-apiserver HA deployments.

This issue is reported at kubernetes/kubernetes#83028 and kubernetes/kubernetes#72102 and a workaround (using wildcard SANs in etcd TLS certificates) is mentioned at kubernetes/kubernetes#72102 (comment). The comments suggest Kubernetes will not have a fix for this until Kubernetes v1.16.3, and that all versions of Kubernetes using etcd3 with TLS certificates (which is kops 1.12 and later, right?) are affected.

Are clusters deployed by kops affected by this issue, or has kops installed a workaround or later etcd3 version?

Please document what is and is not affected and any suggested mitigations.

@rainchei
Copy link

Using kops Version 1.17.0-alpha.1 (git-501baf7e5) and below manifest
(remember to export KOPS_FEATURE_FLAGS=SkipEtcdVersionCheck)

  etcdClusters:
  - name: main
    version: 3.4.3
    etcdMembers:
    - instanceGroup: master-us-east-2a
      name: a
    - instanceGroup: master-us-east-2b
      name: b
    - instanceGroup: master-us-east-2c
      name: c
  - name: events
    version: 3.4.3
    etcdMembers:
    - instanceGroup: master-us-east-2a
      name: a
    - instanceGroup: master-us-east-2b
      name: b
    - instanceGroup: master-us-east-2c
      name: c

we seems to be able to upgrade to etcd 3.4.3

./etcd-manager-ctl -backup-store=<xxx-yyy> get
Backup Store: <xxx-yyy>
member_count:3 etcd_version:"3.4.3"

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 19, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 18, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Development

No branches or pull requests

4 participants