Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the etcd liveness probe in helm chart #110

Closed
bsctl opened this issue Aug 9, 2022 · 0 comments · Fixed by #112
Closed

Improve the etcd liveness probe in helm chart #110

bsctl opened this issue Aug 9, 2022 · 0 comments · Fixed by #112
Assignees
Labels
enhancement New feature or request

Comments

@bsctl
Copy link
Member

bsctl commented Aug 9, 2022

In etcd 3.5, the /health endpoint returns false if there is no etcd leader or leader-election is currently going on. If there is no quorum, etcd will get restart by kubelet but etcd's restart isn't a right behaviour.

Starting from etcd 3.5.3+ support for checking the health of a specific member (as opposed to the cluster as a whole) has been introduced. Upstream kubeadm is switching to using this for static pod health checks, as there's no point in restarting the pod if the cluster as a whole is unhealthy - as a matter of fact it may actually make it worse. We should do the same for Kamaji.

References:

kubernetes/kubernetes#110072
etcd-io/etcd#13399

...
      containers:
        - name: etcd
          image: quay.io/coreos/etcd:v3.5.4
          livenessProbe:
            failureThreshold: 8
            httpGet:
              path: /health?serializable=true
              port: 2381
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 15
          startupProbe:
            failureThreshold: 24
            httpGet:
              path: /health?serializable=true
              port: 2381
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 15
@bsctl bsctl self-assigned this Aug 9, 2022
@bsctl bsctl added the enhancement New feature or request label Aug 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant