[vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434

ajm188 · 2021-02-03T02:54:32Z

Description

This PR extracts some reparent code from methods on the Wrangler struct to top-level functions, and adds tests for them. I'm doing this in advance to allow some code sharing between the legacy and new vtctld grpc APIs, to make those diffs smaller and less risky.

Important: This PR also contains a bugfix in FindValidEmergencyReparentCandidates (formerly (*Wrangler).findValidReparentCandidates) that would cause a non-deterministic panic, depending on what order the
replicationStatusMap is first iterated over in. The panic happens as follows:

Assume you have 2 tablets in the status map. One tablet has a GTID-based relay log position, and the other has a non GTID-based relay log position.

If the GTID tablet comes first, then isGtidBased gets set to a pointer to true, and when we come to the second, non-GTID tablet, we fall through to the 3rd case in the switch (case status.RelayPosition.IsZero()).

If the non-GTID tablet's position is zero, then we return an error, and everything's fine. However, if it's non-zero and non GTID-based, then we successfully make it through the first loop, and later on, we try to FindErrantGTIDs, because *isGtidBased is true. This is where the panic happens, because we give it a mix of Mysql56GTIDSet and non-Mysql56GTIDSet positions.

The fix here is to iterate through the full status map, and not break out early. We track whether we've seen _any_ GTID-based or non-GTID based positions, and fail early accordingly.

Related Issue(s)

Checklist

Should this PR be backported? No
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

…ons, with tests This is for later code sharing between the legacy and new vtctld APIs. This commit also contains a bugfix in `FindValidEmergencyReparentCandidates` / `(*Wrangler).findValidReparentCandidates` that causes a non-deterministic panic, depending on what order the `replicationStatusMap` is first iterated over in. The panic happens as follows: Assume you have 2 tablets in the status map. One tablet has a GTID-based relay log position, and the other has a non GTID-based relay log position. If the GTID tablet comes first, then `isGtidBased` gets set to a pointer to true, and when we come to the second, non-GTID tablet, we fall through to the 3rd case in the switch. If the non-GTID tablet's position is zero, then we return an error, and everything's fine. _However_, if it's non-zero _and_ non-GTID, then we successfully make it through the first loop, and later on, we try to `FindErrantGTIDs`, because `*isGtidBased` is true. This is where the panic happens, because we give it a mix of `Mysql56GTIDSet` and non-`Mysql56GTIDSet` positions. <hr> The fix here is to iterate through the full status map, and not break out early. We track whether we've seen _any_ GTID-based or non-GTID based positions, and fail early accordingly. Signed-off-by: Andrew Mason <amason@slack-corp.com>

…code Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 · 2021-02-03T12:54:30Z

These test failures in wrangler/testlib look legit, and related to my logic change in StopReplicationAndBuildStatusMaps; I'll look into them today:

--- FAIL: TestEmergencyReparentShard (0.03s)
Error:     vtctl_pipe.go:100: I0203 03:03:58.720256 replication.go:170] getting replication position from cell1-0000000001
Error:     vtctl_pipe.go:100: I0203 03:03:58.720358 replication.go:170] getting replication position from cell2-0000000003
Error:     vtctl_pipe.go:100: I0203 03:03:58.720879 replication.go:170] getting replication position from cell1-0000000002
Error:     vtctl_pipe.go:100: I0203 03:03:58.721351 replication.go:170] getting replication position from cell1-0000000000
Error:     vtctl_pipe.go:100: W0203 03:03:58.727724 replication.go:194] failed to get replication status from cell1-0000000000: rpc error: code = Unknown desc = TabletManager.StopReplicationAndGetStatus on cell1-0000000000 error: before status failed: no replication status: before status failed: no replication status
    emergency_reparent_shard_test.go:174: 
        	Error Trace:	emergency_reparent_shard_test.go:174
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = we got a filled-in relay log position, but it's not of type Mysql56GTIDSet, even thought we've determined we need to use GTID based assesment
        	Test:       	TestEmergencyReparentShard

and

--- FAIL: TestEmergencyReparentShardMasterElectNotBest (0.02s)
    emergency_reparent_shard_test.go:280: 
        	Error Trace:	emergency_reparent_shard_test.go:280
        	Error:      	"we got a filled-in relay log position, but it's not of type Mysql56GTIDSet, even thought we've determined we need to use GTID based assesment" does not contain "is not fully caught up"
        	Test:       	TestEmergencyReparentShardMasterElectNotBest

…urn the first error Signed-off-by: Andrew Mason <amason@slack-corp.com>

Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 · 2021-02-03T21:08:43Z

Okay I've changed the strategy for the bugfix to use two booleans, one to keep track if we've ever seen a GTID-based relay log, and one to keep track if we've ever seen a non GTID-based relay log. Separately, we put potential errors onto a FirstErrorRecorder if we see any empty, non GTID-based relay log positions.

Then, we fail if:

we've seen any tablets with both GTID-based and non GTID-based relay logs or
we've seen a tablet with an empty relay log position as well as another tablet with GTID-based relay log positions

> go test -v -race ./go/vt/vtctl/reparentutil/... ./go/vt/wrangler/testlib/... 
ok      vitess.io/vitess/go/vt/vtctl/reparentutil       0.983s
ok      vitess.io/vitess/go/vt/wrangler/testlib 15.413s

ajm188

Okay, @deepthi this should be ready for review now.

ajm188 · 2021-02-03T21:29:13Z

go/vt/vtctl/reparentutil/replication.go

+// FindValidEmergencyReparentCandidates will find candidates for an emergency
+// reparent, and, if successful, return a mapping of those tablet aliases (as
+// raw strings) to their replication positions for later comparison.
+func FindValidEmergencyReparentCandidates(


I'd like to call particular attention to this function in review. It's the only function I deliberately changed the behavior of, when I managed to write a test case that sometimes passed, and sometimes panicked. It's detailed in the PR description, but the tl;dr is that is I believe a mixed set of GTID-based and non GTID-based tablets were not always handled safely, and could result in panics from FindErrantGTIDs, depending on the order of iteration the first time we scan through statusMap to set *isGtidBased in the original implementation.

cc @deepthi

deepthi

Just a couple of nits, otherwise LGTM

deepthi · 2021-02-04T03:01:47Z

go/vt/vtctl/reparentutil/replication.go

+
+ masterStatus, err = tmc.DemoteMaster(groupCtx, tabletInfo.Tablet)
+ if err != nil {
+ msg := "replica %v think it's master but we failed to demote it"


nit: think -> thinks

deepthi · 2021-02-04T03:04:36Z

go/vt/vtctl/reparentutil/replication.go

+ }
+
+ if status.RelayLogPosition.IsZero() {
+ // Potentially bail. If any other tablet hits the non-default


It took me a minute to understand this, it isn't clear what non-default means. Can you rephrase?

Yep, sorry, this was written at the time when I was using a switch case with a default here, and I forgot to go back and update the comment. Will fix.

deepthi · 2021-02-04T03:05:52Z

go/vt/vtctl/reparentutil/util.go

+
+ for _, tablet := range tabletMap {
+ switch {
+ case primaryCell != "" && tablet.Alias.Cell != primaryCell:


This bothered me at first, but I think we looked at it once before and said that by default PRS won't choose a tablet in another cell, but you can always provide it a chosen tablet which will be respected. So I think this is fine.

Yeah, this is a preservation of functionality. Maybe later we can make a vtctld startup flag to turn this off (I'd prefer to be able to PRS cross-cell without having to pick the tablet myself), but definitely far beyond the scope of this change.

deepthi · 2021-02-04T03:06:42Z

cc @PrismaPhonic in case you wanna take a look.

PrismaPhonic · 2021-02-04T09:24:15Z

I'm confused by what real world use case would cause us to have mixed gtidset types.

ajm188 · 2021-02-04T12:49:29Z

I'm confused by what real world use case would cause us to have mixed gtidset types.

We don't support that. The change in implementation here is to detect it correctly and return an error. Previously, there were certain cases where we wouldn't catch it, call status.FindErrantGTIDs with the mixed types, which would cause a panic.

Signed-off-by: Andrew Mason <amason@slack-corp.com>

[vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer Signed-off-by: Richard Bailey <rbailey@slack-corp.com>

ajm188 added 2 commits February 2, 2021 21:44

Update call sites in wrangler to use reparentutil, remove now-dead …

6416147

…code Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 requested a review from deepthi February 3, 2021 02:54

ajm188 requested a review from rohit-nayak-ps as a code owner February 3, 2021 02:54

ajm188 added 2 commits February 3, 2021 15:59

Use a FirstErrorRecorder to process the full statusMap, but later ret…

3dca98e

…urn the first error Signed-off-by: Andrew Mason <amason@slack-corp.com>

Fix typo in error message

b37bb45

Signed-off-by: Andrew Mason <amason@slack-corp.com>

ajm188 commented Feb 3, 2021

View reviewed changes

deepthi reviewed Feb 4, 2021

View reviewed changes

PR feedback: Fix typo in error message, clarify comment

76f3332

Signed-off-by: Andrew Mason <amason@slack-corp.com>

deepthi approved these changes Feb 4, 2021

View reviewed changes

deepthi merged commit b1e4df8 into vitessio:master Feb 4, 2021

askdba added this to the v10.0 milestone Feb 8, 2021

askdba added the Component: Cluster management label Feb 8, 2021

ajm188 deleted the am_reparentutil branch March 4, 2021 16:32

setassociative mentioned this pull request Mar 8, 2021

Port ERS bug fixes into 8.0 tinyspeck/vitess#197

Closed

ajm188 added the Type: Internal Cleanup label May 23, 2021

ajm188 mentioned this pull request Jul 13, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434

[vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434

ajm188 commented Feb 3, 2021

ajm188 commented Feb 3, 2021 •

edited

Loading

ajm188 commented Feb 3, 2021

ajm188 left a comment

ajm188 Feb 3, 2021

deepthi left a comment

deepthi Feb 4, 2021

deepthi Feb 4, 2021 •

edited

Loading

ajm188 Feb 4, 2021

deepthi Feb 4, 2021

ajm188 Feb 4, 2021

deepthi commented Feb 4, 2021

PrismaPhonic commented Feb 4, 2021

ajm188 commented Feb 4, 2021

[vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434

[vtctld / wrangler] Extract some reparent methods out to functions for shared use between wrangler and VtctldServer #7434

Conversation

ajm188 commented Feb 3, 2021

Description

Related Issue(s)

Checklist

Deployment Notes

Impacted Areas in Vitess

ajm188 commented Feb 3, 2021 • edited Loading

ajm188 commented Feb 3, 2021

ajm188 left a comment

Choose a reason for hiding this comment

ajm188 Feb 3, 2021

Choose a reason for hiding this comment

deepthi left a comment

Choose a reason for hiding this comment

deepthi Feb 4, 2021

Choose a reason for hiding this comment

deepthi Feb 4, 2021 • edited Loading

Choose a reason for hiding this comment

ajm188 Feb 4, 2021

Choose a reason for hiding this comment

deepthi Feb 4, 2021

Choose a reason for hiding this comment

ajm188 Feb 4, 2021

Choose a reason for hiding this comment

deepthi commented Feb 4, 2021

PrismaPhonic commented Feb 4, 2021

ajm188 commented Feb 4, 2021

ajm188 commented Feb 3, 2021 •

edited

Loading

deepthi Feb 4, 2021 •

edited

Loading