Relaxes Autopilot promotion logic. #3623
Conversation
When we defaulted the Raft protocol version to 3 in #3477 we made the numPeers() routine more strict to only count voters (this is more conservative and more correct). This had the side effect of breaking rolling updates because it's at odds with the Autopilot non-voter promotion logic. That logic used to wait to only promote to maintain an odd quorum of servers. During a rolling update (add one new server, wait, and then kill an old server) the dead server cleanup would still count the old server as a peer, which is conservative and the right thing to do, and no longer count the non-voter. This would wait to promote, so you could get into a stalemate. It is safer to promote early than remove early, so by promoting as soon as possible we have chosen that as the solution here. Fixes #3611
slackpad
force-pushed
the
relax-autopilot
branch
from
0e977f3
to
f105a6f
Compare
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // fmtServer prints info about a server in a standard way for logging. |
There was a problem hiding this comment.
Sorry I didn't split this out - made some small logging fixes while I was in here.
agent/consul/autopilot_test.go
Outdated
| // be an even quorum. | ||
| if isVoter() { | ||
| t.Fatalf("should not be a voter") | ||
| } |
There was a problem hiding this comment.
Are you sure this test is validating the right thing? Per the new implementation, isVoter should return true after server stabilization period even before you kill one of the old nodes in line 205. I added a sleep before line 200 and this test failed at the isVoter call.
There was a problem hiding this comment.
I think this block can be removed entirely..
There was a problem hiding this comment.
I see what you mean - I was just trying to make sure we saw at least one time where it wasn't a voter and was stabilizing. I'll move up the check to make sure it gets promoted to voter so that's more explicit.
There was a problem hiding this comment.
Actually deleting it is better - we don't need re-verify stabilization here.
When we defaulted the Raft protocol version to 3 in #3477 we made the numPeers() routine more strict to only count voters (this is more conservative and more correct). This had the side effect of
breaking rolling updates because it's at odds with the Autopilot non-voter promotion logic.
That logic used to wait to only promote to maintain an odd quorum of servers. During a rolling update (add one new server, wait, and then kill an old server) the dead server cleanup would still count the old server as a peer, which is conservative and the right thing to do, and no longer count the non-voter. This would wait to promote, so you could get into a stalemate. It is safer to promote early than remove early, so by promoting as soon as possible we have chosen that as the solution here.
Fixes #3611