Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds wait for not ready for external etcd before upgrade #2640

Merged
merged 2 commits into from
Jul 1, 2022

Conversation

jaxesn
Copy link
Member

@jaxesn jaxesn commented Jun 30, 2022

Issue #, if available:

When upgrading a cluster sometimes, it appears to be when the machine is stressed and the bootstrap cluster is running slow, control plane nodes are not upgraded and etcd nodes are left behind. This adds a similar fix to #1542 for external etcd nodes to ensure the cli is waiting for it to upgrade before moving on.

Should fix #2636

Note: @jiayiwang7 is going to finish this by adding tests.

Description of changes:

Testing (if applicable):

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@eks-distro-bot eks-distro-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jun 30, 2022
@codecov
Copy link

codecov bot commented Jun 30, 2022

Codecov Report

Merging #2640 (03568e2) into main (46afe00) will increase coverage by 0.07%.
The diff coverage is 81.81%.

@@            Coverage Diff             @@
##             main    #2640      +/-   ##
==========================================
+ Coverage   57.39%   57.46%   +0.07%     
==========================================
  Files         310      310              
  Lines       25506    25538      +32     
==========================================
+ Hits        14638    14675      +37     
+ Misses       9532     9527       -5     
  Partials     1336     1336              
Impacted Files Coverage Δ
pkg/clustermanager/cluster_manager.go 55.39% <77.77%> (+0.90%) ⬆️
pkg/executables/kubectl.go 42.53% <100.00%> (+0.10%) ⬆️
...ntrollers/controllers/cluster_controller_legacy.go 0.00% <0.00%> (ø)
pkg/workflows/create.go 54.83% <0.00%> (+1.89%) ⬆️
pkg/diagnostics/collectors.go 30.21% <0.00%> (+3.18%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46afe00...03568e2. Read the comment docs.

@jiayiwang7 jiayiwang7 force-pushed the jgw/wait-not-ready-etcd branch from b89e3e6 to 03568e2 Compare July 1, 2022 00:01
@vivek-koppuru vivek-koppuru changed the title WIP: Adds wait for not ready for external etcd before upgrade Adds wait for not ready for external etcd before upgrade Jul 1, 2022
@eks-distro-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@eks-distro-bot eks-distro-bot merged commit 6e6685e into aws:main Jul 1, 2022
@mitalipaygude
Copy link
Member

/cherrypick release-0.10

@eks-distro-pr-bot
Copy link
Contributor

@mitalipaygude: new pull request created: #2676

In response to this:

/cherrypick release-0.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mitalipaygude
Copy link
Member

/cherrypick release-0.10

@eks-distro-pr-bot
Copy link
Contributor

@mitalipaygude: failed to push cherry-picked changes in GitHub: pushing failed, output: "To https://github.com/eks-distro-pr-bot/eks-anywhere\n ! [rejected] cherry-pick-2640-to-release-0.10 -> cherry-pick-2640-to-release-0.10 (non-fast-forward)\nerror: failed to push some refs to 'https://eks-distro-pr-bot:CENSORED@github.com/eks-distro-pr-bot/eks-anywhere'\nhint: Updates were rejected because the tip of your current branch is behind\nhint: its remote counterpart. Integrate the remote changes (e.g.\nhint: 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.\n", error: exit status 1

In response to this:

/cherrypick release-0.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mitalipaygude
Copy link
Member

/cherrypick release-0.10

@eks-distro-pr-bot
Copy link
Contributor

@mitalipaygude: failed to push cherry-picked changes in GitHub: pushing failed, output: "To https://github.com/eks-distro-pr-bot/eks-anywhere\n ! [rejected] cherry-pick-2640-to-release-0.10 -> cherry-pick-2640-to-release-0.10 (non-fast-forward)\nerror: failed to push some refs to 'https://eks-distro-pr-bot:CENSORED@github.com/eks-distro-pr-bot/eks-anywhere'\nhint: Updates were rejected because the tip of your current branch is behind\nhint: its remote counterpart. Integrate the remote changes (e.g.\nhint: 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.\n", error: exit status 1

In response to this:

/cherrypick release-0.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@pokearu
Copy link
Member

pokearu commented Jul 7, 2022

/cherrypick release-0.10

@eks-distro-pr-bot
Copy link
Contributor

@pokearu: failed to push cherry-picked changes in GitHub: pushing failed, output: "To https://github.com/eks-distro-pr-bot/eks-anywhere\n ! [rejected] cherry-pick-2640-to-release-0.10 -> cherry-pick-2640-to-release-0.10 (non-fast-forward)\nerror: failed to push some refs to 'https://eks-distro-pr-bot:CENSORED@github.com/eks-distro-pr-bot/eks-anywhere'\nhint: Updates were rejected because the tip of your current branch is behind\nhint: its remote counterpart. Integrate the remote changes (e.g.\nhint: 'git pull ...') before pushing again.\nhint: See the 'Note about fast-forwards' in 'git push --help' for details.\n", error: exit status 1

In response to this:

/cherrypick release-0.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Control plane nodes not upgraded for same kube version
7 participants