-
Notifications
You must be signed in to change notification settings - Fork 4.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
try to perform a leadership transfer when leaving #11376
Conversation
@@ -0,0 +1,3 @@ | |||
```release-note:enhancement | |||
try to transfer leadership to another server when leaving. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What do you think about something like this?
try to transfer leadership to another server when leaving. | |
raft: Consul leaders will attempt to transfer leadership to another server as part of gracefully leaving the cluster. |
s.logger.Error("failed to remove ourself as a Raft peer", "error", err) | ||
leadershipTransferVersion := version.Must(version.NewVersion(LeaderTransferMinVersion)) | ||
removeServer := false | ||
if ok, _ := ServersInDCMeetMinimumVersion(s, s.config.Datacenter, leadershipTransferVersion); !ok { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be if ok
?
if ok, _ := ServersInDCMeetMinimumVersion(s, s.config.Datacenter, leadershipTransferVersion); !ok { | |
if ok, _ := ServersInDCMeetMinimumVersion(s, s.config.Datacenter, leadershipTransferVersion); ok { |
leadershipTransferVersion := version.Must(version.NewVersion(LeaderTransferMinVersion)) | ||
removeServer := false | ||
if ok, _ := ServersInDCMeetMinimumVersion(s, s.config.Datacenter, leadershipTransferVersion); !ok { | ||
// Transfer leadership to another node then leave the cluster | ||
future := s.raft.LeadershipTransfer() | ||
if err := future.Error(); err != nil { | ||
s.logger.Error("failed to transfer leadership, removing the server", "error", err) | ||
// leadership transfer failed, fallback to removing the server from raft | ||
removeServer = true | ||
} else { | ||
// we are not leader anymore, continue the flow to leave as follower | ||
isLeader = false | ||
} | ||
} else { | ||
// Leadership transfer is not available in the current version, fallback to removing the server from raft | ||
removeServer = true | ||
} | ||
if removeServer { | ||
future := s.raft.RemoveServer(raft.ServerID(s.config.NodeID), 0, 0) | ||
if err := future.Error(); err != nil { | ||
s.logger.Error("failed to remove ourself as raft peer", "error", err) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two high level questions/suggestions:
- Do these comments really help readability? Generally I try to only write docstring comments unless there is something very unusual, and everything here looks pretty expected. Personally I think it reads better without the comments.
- Can we extract any of this to a function to remove the nested
else
statements, and stateful variables?
func (s *Server) leadershipTransfer() (success bool) {
leadershipTransferVersion := version.Must(version.NewVersion(LeaderTransferMinVersion))
ok, _ := ServersInDCMeetMinimumVersion(s, s.config.Datacenter, leadershipTransferVersion)
if !ok {
return false
}
future := s.raft.LeadershipTransfer()
if err := future.Error(); err != nil {
s.logger.Error("failed to transfer leadership, removing the server", "error", err)
return false
}
return true
}
Then in the caller:
if !s.leadershipTransfer() {
future := s.raft.RemoveServer(raft.ServerID(s.config.NodeID), 0, 0)
if err := future.Error(); err != nil {
s.logger.Error("failed to remove ourself as raft peer", "error", err)
}
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The main reason of the comments is because I did not find a good way of removing the nested if else, I will attempt the extract in the new PR.
🍒 If backport labels were added before merging, cherry-picking will start automatically. To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/480806. |
🍒✅ Cherry pick of commit f424faf onto |
* try to perform a leadership transfer when leaving * add a changelog
This reverts commit f424faf.
🍒 If backport labels were added before merging, cherry-picking will start automatically. To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/481009. |
🍒✅ Cherry pick of commit f424faf onto |
* try to perform a leadership transfer when leaving * add a changelog
This is a port of #11242 (1.8.x) for main