raft: provide protection against unbounded Raft log growth #10167

nvanbenschoten · 2018-10-09T16:51:10Z

The suggested pattern for Raft proposals is that they be retried
periodically until they succeed. This turns out to be an issue
when a leader cannot commit entries because the leader will continue
to append re-proposed entries to its log without committing anything.
This can result in the uncommitted tail of a leader's log growing
without bound until it is able to commit entries.

This change add a safeguard to protect against this case where a
leader's log can grow without bound during loss of quorum scenarios.
It does so by introducing a new, optional MaxUncommittedEntries
configuration. This config limits the max number of uncommitted
entries that may be appended to a leader's log. Once this limit
is exceeded, proposals will begin to return ErrProposalDropped
errors.

See cockroachdb/cockroach#27772

bdarnell · 2018-10-09T17:17:46Z

raft/raft.go

+ // MaxUncommittedEntries limits the max number of uncommitted entries that
+ // may be appended to a leader's log. Once this limit is exceeded, proposals
+ // will begin to return ErrProposalDropped errors. Note: 0 for no limit.
+ MaxUncommittedEntries int


I think I'd rather make this about the total size of the entries instead of just the count. (Maybe with a provision that at least one uncommitted entry will always be allowed regardless of its size)

Basing this off the aggregate entry size is tricky. It means we'll need to keep track of the uncommitted entry size as entries are proposed and committed and probably also compute this on leadership changes, which would effectively undo #9073.

Also, for what it's worth, other flow control mechanisms like MaxInflightMsgs operate off message count instead of message size, so there is precendence for this approach.

Well, if the size of the log were bounded, #9073 wouldn't have been necessary (or at least not as critical). Also, an approximation here is reasonable: only track the uncommitted log size within a single term; assume that the uncommitted size is zero on becoming leader.

I don't feel too strongly about this given the precedent of MaxInflightMsgs, but if I could go back in time I think I'd change that one to count bytes too.

I made the change, but I don't know how I feel about it. I think it's actually harder to pick a reasonable limit based on size than it is is based on entry count. We want the limit to catch runaway proposals without ever affecting normal operation. What size do we set this to to ensure that? A multiple of the max proposal size? An arbitrary large value like 128MB? If we do that then the log can still grow to millions of small entries. A smaller value is no better because then we start dropping large proposals pretty quickly.

Maybe the argument is that a million small entries is fine as long as they remain under a certain byte size.

Yes, my argument is that a million small entries are fine.

You haven't pushed your branch with this change.

Done. PTAL.

bdarnell · 2018-10-09T17:22:52Z

raft/raft_test.go

+ r.becomeLeader()
+
+ // Set the two followers to the replicate state. Commit to tail of log.
+ const followers = 2


s/followers/numFollowers/

bdarnell · 2018-10-09T17:28:31Z

It might be a good idea to make a separate error for this (in retrospect ErrProposalDropped should have been a type with a reason field instead of a singleton). The ideal retry behavior depends on whether the proposal was dropped because of no leader or because the log was full.

codecov-io · 2018-10-09T19:50:01Z

Codecov Report

Merging #10167 into master will increase coverage by 0.15%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #10167      +/-   ##
==========================================
+ Coverage   71.54%    71.7%   +0.15%     
==========================================
  Files         390      390              
  Lines       36319    36346      +27     
==========================================
+ Hits        25986    26063      +77     
+ Misses       8518     8472      -46     
+ Partials     1815     1811       -4

Impacted Files	Coverage Δ
raft/rawnode.go	`68.59% <100%> (+0.26%)`	⬆️
raft/node.go	`92.74% <100%> (+1.56%)`	⬆️
raft/raft.go	`91.63% <100%> (+0.28%)`	⬆️
pkg/logutil/zap_grpc.go	`47.61% <0%> (-4.77%)`	⬇️
etcdserver/api/v3rpc/watch.go	`80.06% <0%> (-3.93%)`	⬇️
pkg/testutil/recorder.go	`77.77% <0%> (-3.71%)`	⬇️
proxy/grpcproxy/watch.go	`88.55% <0%> (-3.02%)`	⬇️
clientv3/concurrency/election.go	`79.52% <0%> (-2.37%)`	⬇️
clientv3/watch.go	`90.25% <0%> (-1.91%)`	⬇️
etcdserver/raft.go	`78.67% <0%> (-1.43%)`	⬇️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3c6c05b...73c20cc. Read the comment docs.

nvanbenschoten · 2018-10-11T22:14:52Z

It might be a good idea to make a separate error for this (in retrospect ErrProposalDropped should have been a type with a reason field instead of a singleton). The ideal retry behavior depends on whether the proposal was dropped because of no leader or because the log was full.

I don't feel particularly strongly about this, but for what it's worth there are already 5 cases that use ErrProposalDropped and not all of them have to do with an unknown leader. The error description is certainly broad enough to apply here.

bdarnell

LGTM.

I'm not sure what's going on in that travis-ci failure. @xiang90 or @gyuho, any thoughts?

gyuho · 2018-10-12T16:46:44Z

raft/raft_test.go

- nw.drop(one, other, 2.0) // always drop
- nw.drop(other, one, 2.0) // always drop
+ nw.drop(one, other, 1.0) // always drop
+ nw.drop(other, one, 1.0) // always drop


Why do we change this?

raft/raft_test.go:4201:42: (*network).drop - perc always receives 1.0 (1)

Seems like now we only use 1.0 for nw.drop (CI was complaining :0)

It was a fix for what I thought was a typo. Dropping 200% of messages doesn't really make sense.

I didn't think it would trigger a lint warning. Reverted.

xiang90 · 2018-10-12T21:27:53Z

raft/raft.go

+ // entries that may be appended to a leader's log. Once this limit is
+ // exceeded, proposals will begin to return ErrProposalDropped errors.
+ // Note: 0 for no limit.
+ MaxUncommittedEntriesSize uint64


what is the unit of the size?

xiang90 · 2018-10-12T21:28:39Z

@nvanbenschoten LGTM. After fixing the nit.

xiang90 · 2018-10-12T21:37:55Z

@nvanbenschoten we also need to update https://github.com/etcd-io/etcd/tree/master/raft#features to include this feature.

nvanbenschoten · 2018-10-12T21:54:05Z

we also need to update https://github.com/etcd-io/etcd/tree/master/raft#features to include this feature.

Done. Thanks for the reviews.

xiang90 · 2018-10-12T22:05:23Z

LGTM. Will merge when CI passes.

nvanbenschoten · 2018-10-12T23:31:14Z

CI looks flaky. Anything you need me to do?

xiang90 · 2018-10-12T23:37:02Z

raft/raft.go

@@ -215,6 +220,12 @@ func (c *Config) validate() error {
 return errors.New("storage cannot be nil")
 }

+ if c.MaxUncommittedEntriesSize < 0 {


@nvanbenschoten CI failed due to this checking.

raft/raft.go:223:5: unsigned values are never < 0 (SA4003)

Fixed. CI is passing now.

The suggested pattern for Raft proposals is that they be retried periodically until they succeed. This turns out to be an issue when a leader cannot commit entries because the leader will continue to append re-proposed entries to its log without committing anything. This can result in the uncommitted tail of a leader's log growing without bound until it is able to commit entries. This change add a safeguard to protect against this case where a leader's log can grow without bound during loss of quorum scenarios. It does so by introducing a new, optional ``MaxUncommittedEntriesSize configuration. This config limits the max aggregate size of uncommitted entries that may be appended to a leader's log. Once this limit is exceeded, proposals will begin to return ErrProposalDropped errors. See cockroachdb/cockroach#27772

This picks up etcd-io/etcd#10167.

Picks up etcd-io/etcd#10167. Future commits will use the new setting to replace broken logic that prevented unbounded Raft log growth. This also picks up etcd-io/etcd#10063. Release note: None

…rotection This change reverts most of the non-testing code from 03b116f and f2f3fd2 and replaces it with use of the MaxUncommittedEntriesSize config. This configuration was added in etcd-io/etcd#10167 and provides protection against unbounded Raft log growth when a Raft group stops being able to commit entries. It makes proposals into Raft safer because proposers don't need to verify before the fact that the proposal isn't a duplicate that might be blowing up the size of the Raft group. By default, the configuration is set to double the Replica's proposal quota. The logic here is that the quotaPool should be responsible for throttling proposals in all cases except for unbounded Raft re-proposals because it queues efficiently instead of dropping proposals on the floor indiscriminately. Release note (bug fix): Fix a bug where Raft proposals could get stuck if forwarded to a leader who could not itself append a new entry to its log.

This picks up etcd-io/etcd#10167.

Picks up etcd-io/etcd#10167. Future commits will use the new setting to replace broken logic that prevented unbounded Raft log growth. This also picks up etcd-io/etcd#10063. Release note: None

…rotection This change reverts most of the non-testing code from 03b116f and f2f3fd2 and replaces it with use of the MaxUncommittedEntriesSize config. This configuration was added in etcd-io/etcd#10167 and provides protection against unbounded Raft log growth when a Raft group stops being able to commit entries. It makes proposals into Raft safer because proposers don't need to verify before the fact that the proposal isn't a duplicate that might be blowing up the size of the Raft group. By default, the configuration is set to double the Replica's proposal quota. The logic here is that the quotaPool should be responsible for throttling proposals in all cases except for unbounded Raft re-proposals because it queues efficiently instead of dropping proposals on the floor indiscriminately. Release note (bug fix): Fix a bug where Raft proposals could get stuck if forwarded to a leader who could not itself append a new entry to its log.

31408: storage: replace remote proposal tracking with uncommitted log size protection r=nvanbenschoten a=nvanbenschoten Closes #30064. This change reverts most of the non-testing code from 03b116f and f2f3fd2 and replaces it with use of the MaxUncommittedEntriesSize config. This configuration was added in etcd-io/etcd#10167 and provides protection against unbounded Raft log growth when a Raft group stops being able to commit entries. It makes proposals into Raft safer because proposers don't need to verify before the fact that the proposal isn't a duplicate that might be blowing up the size of the Raft group. By default, the configuration is set to double the Replica's proposal quota. The logic here is that the quotaPool should be responsible for throttling proposals in all cases except for unbounded Raft re-proposals because it queues efficiently instead of dropping proposals on the floor indiscriminately. Release note (bug fix): Fix a bug where Raft proposals could get stuck if forwarded to a leader who could not itself append a new entry to its log. This will be backported, but not to 2.1.0. The plan is to get it into 2.1.1. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>

Picks up etcd-io/etcd#10167. Future commits will use the new setting to replace broken logic that prevented unbounded Raft log growth. This also picks up etcd-io/etcd#10063. Release note: None

…rotection This change reverts most of the non-testing code from 03b116f and f2f3fd2 and replaces it with use of the MaxUncommittedEntriesSize config. This configuration was added in etcd-io/etcd#10167 and provides protection against unbounded Raft log growth when a Raft group stops being able to commit entries. It makes proposals into Raft safer because proposers don't need to verify before the fact that the proposal isn't a duplicate that might be blowing up the size of the Raft group. By default, the configuration is set to double the Replica's proposal quota. The logic here is that the quotaPool should be responsible for throttling proposals in all cases except for unbounded Raft re-proposals because it queues efficiently instead of dropping proposals on the floor indiscriminately. Release note (bug fix): Fix a bug where Raft proposals could get stuck if forwarded to a leader who could not itself append a new entry to its log.

This picks up etcd-io/etcd#10167.

nvanbenschoten mentioned this pull request Oct 9, 2018

teamcity: failed test: bank/node-restart cockroachdb/cockroach#30064

Closed

bdarnell approved these changes Oct 9, 2018

View reviewed changes

nvanbenschoten force-pushed the nvanbenschoten/limitUncommitted branch from 11f4f21 to 5e7d14c Compare October 9, 2018 17:33

siddontang mentioned this pull request Oct 10, 2018

raft: provide protection against unbounded Raft log growth tikv/raft-rs#131

Closed

nvanbenschoten force-pushed the nvanbenschoten/limitUncommitted branch from 5e7d14c to 4470b3d Compare October 11, 2018 22:12

bdarnell approved these changes Oct 12, 2018

View reviewed changes

gyuho reviewed Oct 12, 2018

View reviewed changes

nvanbenschoten force-pushed the nvanbenschoten/limitUncommitted branch from 4470b3d to 8276a9f Compare October 12, 2018 19:08

xiang90 reviewed Oct 12, 2018

View reviewed changes

nvanbenschoten force-pushed the nvanbenschoten/limitUncommitted branch from 8276a9f to 697dddf Compare October 12, 2018 21:53

gyuho added the Release-Note label Oct 12, 2018

xiang90 reviewed Oct 12, 2018

View reviewed changes

nvanbenschoten added 2 commits October 13, 2018 23:25

raft: Fix spelling in doc.go

7be7ac5

nvanbenschoten force-pushed the nvanbenschoten/limitUncommitted branch from 697dddf to f894f32 Compare October 14, 2018 03:25

raft: Fix comment on sendHeartbeat

73c20cc

nvanbenschoten force-pushed the nvanbenschoten/limitUncommitted branch from f894f32 to 73c20cc Compare October 14, 2018 04:03

xiang90 merged commit dac8c6f into etcd-io:master Oct 14, 2018

nvanbenschoten added a commit to cockroachdb/vendored that referenced this pull request Oct 14, 2018

Bump go.etcd.io/etcd

e858131

This picks up etcd-io/etcd#10167.

nvanbenschoten mentioned this pull request Oct 15, 2018

storage: replace remote proposal tracking with uncommitted log size protection cockroachdb/cockroach#31408

Merged

nvanbenschoten mentioned this pull request Oct 16, 2018

storage: divergent entry application can lead to request stalls on Range splits cockroachdb/cockroach#31330

Closed

nvanbenschoten added a commit to cockroachdb/vendored that referenced this pull request Oct 17, 2018

Bump go.etcd.io/etcd

ca4276c

This picks up etcd-io/etcd#10167.

This was referenced Oct 18, 2018

storage: sideloaded Raft entries don't play well with Raft memory limits cockroachdb/cockroach#31618

Closed

raft: fix bug in unbounded log growth prevention mechanism #10199

Merged

nvanbenschoten mentioned this pull request Nov 26, 2018

release-2.1: storage: replace remote proposal tracking with uncommitted log size protection cockroachdb/cockroach#32600

Merged

nvanbenschoten added a commit to cockroachdb/vendored that referenced this pull request Nov 26, 2018

Bump go.etcd.io/etcd

e564efe

This picks up etcd-io/etcd#10167.

tbg mentioned this pull request Dec 18, 2018

roachtest: import/tpch/nodes=8 failed [release-2.1] cockroachdb/cockroach#32898

Closed

absolute8511 mentioned this pull request Mar 13, 2019

Consider merge some optimize from etcd-raft youzan/ZanRedisDB#23

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raft: provide protection against unbounded Raft log growth #10167

raft: provide protection against unbounded Raft log growth #10167

nvanbenschoten commented Oct 9, 2018

bdarnell Oct 9, 2018

nvanbenschoten Oct 9, 2018

bdarnell Oct 9, 2018

nvanbenschoten Oct 9, 2018

bdarnell Oct 10, 2018

nvanbenschoten Oct 11, 2018

bdarnell Oct 9, 2018

nvanbenschoten Oct 9, 2018

bdarnell commented Oct 9, 2018

codecov-io commented Oct 9, 2018 •

edited

Loading

nvanbenschoten commented Oct 11, 2018

bdarnell left a comment

gyuho Oct 12, 2018

nvanbenschoten Oct 12, 2018

xiang90 Oct 12, 2018

nvanbenschoten Oct 12, 2018

xiang90 commented Oct 12, 2018

xiang90 commented Oct 12, 2018

nvanbenschoten commented Oct 12, 2018

xiang90 commented Oct 12, 2018

nvanbenschoten commented Oct 12, 2018

xiang90 Oct 12, 2018

nvanbenschoten Oct 14, 2018

raft: provide protection against unbounded Raft log growth #10167

raft: provide protection against unbounded Raft log growth #10167

Conversation

nvanbenschoten commented Oct 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdarnell commented Oct 9, 2018

codecov-io commented Oct 9, 2018 • edited Loading

Codecov Report

nvanbenschoten commented Oct 11, 2018

bdarnell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiang90 commented Oct 12, 2018

xiang90 commented Oct 12, 2018

nvanbenschoten commented Oct 12, 2018

xiang90 commented Oct 12, 2018

nvanbenschoten commented Oct 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Oct 9, 2018 •

edited

Loading