Improve join logic to handle unreachable nodes #560
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bschimke95
changed the title
bschimke95
force-pushed
the
bschimke95/debug-upgrade-race-condition
branch
from
766c1ea to
5d406dc
Compare
neoaggelos
reviewed
Jul 22, 2024
bschimke95
force-pushed
the
bschimke95/debug-upgrade-race-condition
branch
from
71a8fa4 to
55382de
Compare
|
rebased on top of #562 to verify this fix works. |
louiseschmidtgen
approved these changes
Jul 30, 2024
Co-authored-by: Mateo Florido <32885896+mateoflorido@users.noreply.github.com>
bschimke95
force-pushed
the
bschimke95/upgrade-microcluster
branch
from
2ab601e to
af9e3ac
Compare
Co-authored-by: Louise K. Schmidtgen <louise.schmidtgen@canonical.com>
bschimke95
force-pushed
the
bschimke95/debug-upgrade-race-condition
branch
from
b172853 to
bafed4c
Compare
neoaggelos
reviewed
Jul 31, 2024
Comment on lines
+73
to
+83
| // Get remote certificate from the cluster member. We only need one node to be reachable for this. | ||
| // One might fail because the node is not part of the cluster anymore but was at the time the token was created. | ||
| var cert *x509.Certificate | ||
| var address string | ||
| var err error | ||
| for _, address = range token.JoinAddresses { | ||
| cert, err = utils.GetRemoteCertificate(address) | ||
| if err == nil { | ||
| break | ||
| } | ||
| } |
There was a problem hiding this comment.
In general, try to avoid mixing dependency upgrades along with changes in the code logic, as it would be harder to backport and/or revert if required.
bschimke95
added a commit
that referenced
this pull request
Aug 5, 2024
--------- Co-authored-by: Mateo Florido <32885896+mateoflorido@users.noreply.github.com> Co-authored-by: Louise K. Schmidtgen <louise.schmidtgen@canonical.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes an issue where the join process would fail if a node in the join_address list is unreachable.
When a token is created, it includes the IP addresses of all nodes in the cluster. If a node is removed between the token creation and joining, the join process previously failed because it tried to connect to the non-existent node.
Now, the client discovery logic will continue attempting to connect to other nodes in the join_address list instead of failing early. This ensures that the join process can succeed as long as at least one node is reachable.
This PR fixes this issue on
k8s-snaplevel for the worker nodes and requires the microcluster upgrades from #562 to actually pass the integration test.