Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add validations and logs to fix common issues faced during MoveTables/Reshard #6814

Merged
merged 8 commits into from
Oct 6, 2020

Conversation

rohit-nayak-ps
Copy link
Contributor

@rohit-nayak-ps rohit-nayak-ps commented Oct 3, 2020

Workflow artifacts like entries in the _vt.vreplication and _vt.resharding_journal tables, if not cleaned up, after a workflow is completed can cause strange errors during subsequent workflows. DropSources clears these, but users don't always run DropSources. Added validations include:

  • a check if previous _vt.vreplication entries exist in frozen state
  • a check if resharding journals from a previous reshard are present on the current target

Also

  • add tables to the VSchema during movetables for unsharded keyspace targets (some workflows fail with a "table not found"
    if user tries to reshard it and has only specified a csv table list to the movetables)
  • more error logs for workflows for debugging purposes. Currently errors are just returned to the grpc client and hence not logged in the tablet logs

Signed-off-by: Rohit Nayak rohit@planetscale.com

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
…complete workflows

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
… errors. 2 tests skipped for now.

Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@rohit-nayak-ps rohit-nayak-ps changed the title WIP: Add validations and logs to fix common issues faced by early vrep adopters WIP: Add validations and logs to fix common issues faced by initial vrep users Oct 3, 2020
@rohit-nayak-ps rohit-nayak-ps changed the title WIP: Add validations and logs to fix common issues faced by initial vrep users WIP: Add validations and logs to fix common issues faced during MoveTables/Reshard Oct 3, 2020
@rohit-nayak-ps rohit-nayak-ps changed the title WIP: Add validations and logs to fix common issues faced during MoveTables/Reshard Add validations and logs to fix common issues faced during MoveTables/Reshard Oct 3, 2020
@rohit-nayak-ps rohit-nayak-ps marked this pull request as ready for review October 3, 2020 21:58
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Copy link
Member

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we check at the start of a Reshard workflow whether a non-empty sharded vschema has been provided?

go/vt/wrangler/fake_dbclient_test.go Outdated Show resolved Hide resolved
go/vt/wrangler/fake_dbclient_test.go Outdated Show resolved Hide resolved
go/vt/wrangler/materializer_env_test.go Outdated Show resolved Hide resolved
go/vt/wrangler/fake_dbclient_test.go Outdated Show resolved Hide resolved
go/vt/wrangler/traffic_switcher_test.go Show resolved Hide resolved
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
@rohit-nayak-ps
Copy link
Contributor Author

Do we check at the start of a Reshard workflow whether a non-empty sharded vschema has been provided?

There is a check for an existing vschema, doesn't check if it is sharded. I will review that and add it when I fix a bunch of pending resharding issues.

@@ -145,12 +150,12 @@ func (dc *fakeDBClient) verifyQueries(t *testing.T) {
t.Helper()
for query, dbrs := range dc.queries {
if !dbrs.exhausted() {
t.Errorf("query: %v has unreturned results", query)
assert.FailNow(t, "expected query: %v did not get executed during the test", query)
}
}
for query, dbrs := range dc.queriesRE {
if !dbrs.exhausted() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be rewritten as

assert.True(t, dbrs.exhausted(), "message")

but we won't hold up this PR for that.

@deepthi deepthi merged commit a0a72cb into vitessio:master Oct 6, 2020
@deepthi deepthi deleted the rn-workflow-validations branch October 6, 2020 17:46
@askdba askdba added this to the v8.0 milestone Oct 7, 2020
@rohit-nayak-ps rohit-nayak-ps linked an issue Oct 15, 2020 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Reshard: table /.* not found in vschema
3 participants