Skip to content

release-23.2: kvserver: remove changed replicas in purgatory from replica set #115037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kvoli merged 1 commit into from
Nov 28, 2023
Merged

release-23.2: kvserver: remove changed replicas in purgatory from replica set #115037

kvoli merged 1 commit into from
Nov 28, 2023

Conversation

@blathers-crl
Copy link

@blathers-crl blathers-crl bot commented Nov 23, 2023 •
edited by kvoli
Loading

Backport 1/1 commits from #114365 on behalf of @kvoli.

/cc @cockroachdb/release

It was possible for a replica to be stuck processing in a queue's
replica set. This could occur when a replica had recently been removed
from purgatory for processing but was destroyed, or replica ID changed
before being processed.

When this occurred, the replica could never be processed by the queue
again, potentially leading to decommission stalls, constraint violations
or under(over)replication.

Remove the replica from the queue set upon encountering a replica which
was destroyed, or replica ID changed when processing purgatory. This
prevents the replica from becoming stuck in a processing state in the
queue set.

Fixes: #112761
Fixes: #110761

Release note (bug fix): The store queues will no longer leave purgatory
replicas which have changed replica IDs, or have been destroyed stuck
unable to process via the respective queue again if re-added.

Release justification: Fixes serious bug.

@kvoli
kvserver: remove changed replicas in purgatory from replica set
c567b30 
It was possible for a replica to be stuck processing in a queue's
replica set. This could occur when a replica had recently been removed
from purgatory for processing but was destroyed, or replica ID changed
before being processed.

When this occurred, the replica could never be processed by the queue
again, potentially leading to decommission stalls, constraint violations
or under(over)replication.

Remove the replica from the queue set upon encountering a replica which
was destroyed, or replica ID changed when processing purgatory. This
prevents the replica from becoming stuck in a processing state in the
queue set.

Fixes: #112761
Fixes: #110761

Release note (bug fix): The store queues will no longer leave purgatory
replicas which have changed replica IDs, or have been destroyed stuck
unable to process via the respective queue again if re-added.
@blathers-crl blathers-crl bot requested a review from a team as a code owner November 23, 2023 14:39
@blathers-crl blathers-crl bot force-pushed the blathers/backport-release-23.2-114365 branch from 814dec5 to 8cb6c1e Compare November 23, 2023 14:39
@blathers-crl blathers-crl bot added blathers-backport This is a backport that Blathers created automatically. O-robot Originated from a bot. labels Nov 23, 2023
@blathers-crl blathers-crl bot force-pushed the blathers/backport-release-23.2-114365 branch from 1f4ae2c to c567b30 Compare November 23, 2023 14:39
@blathers-crl
Copy link
Author

blathers-crl bot commented Nov 23, 2023 •
edited by kvoli
Loading

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Backports should only be created for serious
    issues     or test-only changes.
  • Backports should not break backwards-compatibility.
  • Backports should change as little code as possible.
  • Backports should not change on-disk formats or node communication protocols.
  • Backports should not add new functionality (except as defined
    here).
  • Backports must not add, edit, or otherwise modify cluster versions; or add version gates.
  • All backports must be reviewed by the owning areas TL and one additional
    TL. For more information as to how that review should be conducted, please consult the backport
    policy    .
If your backport adds new functionality, please ensure that the following additional criteria are satisfied:
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters. State changes must be further protected such that nodes running old binaries will not be negatively impacted by the new state (with a mixed version test added).
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.
  • Your backport must be accompanied by a post to the appropriate Slack
    channel (#db-backports-point-releases or #db-backports-XX-X-release) for awareness and discussion.

Also, please add a brief release justification to the body of your PR to justify this
backport.

@blathers-crl blathers-crl bot requested a review from andrewbaptist November 23, 2023 14:39
@blathers-crl blathers-crl bot added the backport Label PR's that are backports to older release branches label Nov 23, 2023
@blathers-crl blathers-crl bot requested review from kvoli and nvb November 23, 2023 14:39
@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Nov 23, 2023

This change is Reviewable

@kvoli kvoli requested a review from jbowens November 27, 2023 15:46
jbowens
jbowens approved these changes Nov 27, 2023
View reviewed changes
Copy link
Collaborator

@jbowens jbowens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did not review for correctness but secondary backport policy review lgtm

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @andrewbaptist, @kvoli, and @nvanbenschoten)

andrewbaptist
andrewbaptist approved these changes Nov 27, 2023
View reviewed changes
Copy link

@andrewbaptist andrewbaptist left a comment •
edited by cockroach-dev-inf
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

@kvoli
Copy link
Contributor

kvoli commented Nov 28, 2023

TYFTRs!

TestSQLStatsFlush stress race timed out waiting for full replication, doesn't look related.

Details 

goroutine 53121 [runnable]:
bytes.Equal(...)
	GOROOT/src/bytes/bytes.go:20
github.com/cockroachdb/redact/internal/escape.InternalEscapeBytes({0xc00925a000, 0x1d3e, 0x1dbb}, 0x3, 0x1, 0x0)
	github.com/cockroachdb/redact/internal/escape/external/com_github_cockroachdb_redact/internal/escape/escape.go:96 +0x285
github.com/cockroachdb/redact/internal/buffer.(*Buffer).escapeToEnd(...)
	github.com/cockroachdb/redact/internal/buffer/external/com_github_cockroachdb_redact/internal/buffer/buffer.go:239
github.com/cockroachdb/redact/internal/buffer.(*Buffer).SetMode(0xc00a25ef00, 0x1)
	github.com/cockroachdb/redact/internal/buffer/external/com_github_cockroachdb_redact/internal/buffer/buffer.go:255 +0x105
github.com/cockroachdb/redact/internal/rfmt.restorer.restore({0xc00a25ef00?, 0xc009246000?, 0x1d3b?})
	github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/helpers.go:80 +0x45
github.com/cockroachdb/redact/internal/rfmt.(*pp).fmtString(0xc00a25ef00, {0xc009246000, 0x1d3b}, 0x73)
	github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:484 +0x43a
github.com/cockroachdb/redact/internal/rfmt.(*pp).handleMethods(0xc00a25ef00, 0x73)
	github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:690 +0x9f5
github.com/cockroachdb/redact/internal/rfmt.(*pp).printArg(0xc00a25ef00, {0x78f70a0, 0xc015173980}, 0x73)
	github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:810 +0x156a
github.com/cockroachdb/redact/internal/rfmt.(*pp).doPrintf(0xc00a25ef00, {0x7bbafb5, 0x2f}, {0xc00e76f1a8?, 0x1, 0x1})
	github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:1144 +0x35a
github.com/cockroachdb/redact/internal/rfmt.Fprintf({0xb155da0, 0xc01874a150}, {0x7bbafb5, 0x2f}, {0xc00e76f1a8, 0x1, 0x1})
	github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:212 +0x8e
github.com/cockroachdb/redact/builder.(*StringBuilder).Printf(0xc01874a150, {0x7bbafb5, 0x2f}, {0xc00e76f1a8, 0x1, 0x1})
	github.com/cockroachdb/redact/builder/external/com_github_cockroachdb_redact/builder/builder.go:100 +0x85
github.com/cockroachdb/cockroach/pkg/util/log.makeUnstructuredEntry({_, _}, _, _, _, _, {_, _}, {0xc00e76f1a8, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/log_entry.go:245 +0x1e5
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0xb18b0c0, 0xc005dd2410}, 0x0, 0x1, 0x74b7c40?, 0x0?, {0x7bbafb5, 0x2f}, {0xc00e76f1a8, 0x1, ...})
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:217 +0x1da
github.com/cockroachdb/cockroach/pkg/util/log.VEventf(...)
	github.com/cockroachdb/cockroach/pkg/util/log/trace.go:238
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).DelegateSnapshot(0xc00fc801c0, {0xb18b0c0, 0xc005dd2410}, 0xc01986bb80)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:1198 +0x405
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendSnapshotUsingDelegate.func2({0xb18b0c0, 0xc005dd2410})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:2875 +0x90
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0xb18b168, 0xc019de3a70}, {0x7ae8016, 0xd}, 0x34630b8a000, 0xc00e76f608)
	github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x128
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendSnapshotUsingDelegate(0xc00b5d9600, {0xb18b168, 0xc019de3a70}, {0x2, 0x2, 0x3, 0x1}, 0x1, 0x1, 0x1, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:2872 +0xe65
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).initializeRaftLearners(0xc00b5d9600, {0xb18b168, 0xc019de3a70}, 0xc00e49fdc0, 0x0?, 0x0?, 0x0?, {0x7b10b1b, 0x16}, {0xc00724e5f0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:1846 +0xdb6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).changeReplicasImpl(0xc00b5d9600, {0xb18b168, 0xc019de3a70}, 0x0?, 0x0?, 0x0?, 0xb18b168?, {0x7b10b1b, 0x16}, {0xc00724e5f0, ...}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:1083 +0x87d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).changeReplicas(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:1114
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).applyChange(0xc004d6ab40?, {0xb18b168, 0xc019de3a70}, {0x4, {0xb1d8928, 0xc00b5d9600}, {0xb1673e0, 0xc00d55c780}, {0x1, 0x1, ...}}, ...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:815 +0x22e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChange(0xc00de75680, {0xb18b168, 0xc019de3a70}, 0x7aeffb0?, 0xf?, 0xc00e7706b0?, 0x1?, 0x50?, 0x0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:916 +0x358
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).processOneChangeWithTracing(0xc00de75680, {0xb18b130, 0xc019de39e0}, 0xc00b5d9600, 0x4000000000000000?, 0x5?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:759 +0x27a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicateQueue).process(0xc00de75680, {0xb18b130, 0xc019de39e0}, 0xc00b5d9600, {0x7f7261889358, 0xc017a83180})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replicate_queue.go:668 +0x48f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1({0xb18b130, 0xc019de39e0})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1036 +0x2f5
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0xb18b168, 0xc019de39b0}, {0xc007160660, 0x22}, 0xdf8475800, 0xc00e7711c0)
	github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x128
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica(0xc00cfac6e0, {0xb18b168, 0xc019de3980}, {0xb1d3560, 0xc00b5d9600})
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:995 +0x65c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).DrainQueue(0xc00cfac6e0, 0xc00e5cd348?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:1406 +0x205
github.com/cockroachdb/cockroach/pkg/kv/kvserver.forceScanAndProcess({0xb18b0f8, 0xc0001b2018}, 0xc00d036a80, 0xc00cfac6e0)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue_helpers_testutil.go:42 +0x15e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).ForceReplicationScanAndProcess(...)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue_helpers_testutil.go:55
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication.func2(0xc00d036a80)
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1448 +0x150
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores.func1(0x7361cc0?, 0xc01fc7e6c0?)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:149 +0x5e
github.com/cockroachdb/cockroach/pkg/util/syncutil.(*IntMap).Range(0xc00357ba20, 0xc00e771748)
	github.com/cockroachdb/cockroach/pkg/util/syncutil/int_map.go:358 +0x191
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).VisitStores(0xc00357b9e0, 0xc00e5cd850)
	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:148 +0x76
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).WaitForFullReplication(0xc010146300)
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1440 +0x4ba
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start(0xc010146300, {0xb1c8510?, 0xc000a5a1a0?})
	github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:449 +0xa0b
github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster({_, _}, _, {{{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...}, ...})
	github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:288 +0x145
pkg/sql/sqlstats/persistedsqlstats/persistedsqlstats_test_test.TestSQLStatsFlush(0xc000a5a1a0)
	pkg/sql/sqlstats/persistedsqlstats/persistedsqlstats_test_test/pkg/sql/sqlstats/persistedsqlstats/flush_test.go:75 +0x36d
testing.tRunner(0xc000a5a1a0, 0x7e24bb8)
	GOROOT/src/testing/testing.go:1576 +0x217
created by testing.(*T).Run
	GOROOT/src/testing/testing.go:1629 +0x806

@kvoli kvoli merged commit a90c3b0 into release-23.2 Nov 28, 2023
@kvoli kvoli deleted the blathers/backport-release-23.2-114365 branch November 28, 2023 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jbowens jbowens jbowens approved these changes

@kvoli kvoli Awaiting requested review from kvoli

+2 more reviewers

@nvb nvb nvb approved these changes

@andrewbaptist andrewbaptist andrewbaptist approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kvoli kvoli

Labels

backport Label PR's that are backports to older release branches blathers-backport This is a backport that Blathers created automatically. O-robot Originated from a bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@cockroach-teamcity @kvoli @jbowens @nvb @andrewbaptist