add vtgate flag that explicitly allows vstream copy #125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
pbibra
commented
Sep 11, 2023
| @@ -540,6 +545,12 @@ func (vs *vstream) streamFromTablet(ctx context.Context, sgtid *binlogdatapb.Sha | |||
| log.Infof("Starting to vstream from %s", tablet.Alias.String()) | |||
| // Safe to access sgtid.Gtid here (because it can't change until streaming begins). | |||
| var vstreamCreatedOnce sync.Once | |||
|
|
|||
| if !vs.vsm.allowVstreamCopy && (sgtid.Gtid == "" || len(sgtid.TablePKs) > 0) { | |||
There was a problem hiding this comment.
verifying whether this condition is sufficient for identifying a vcopy
There was a problem hiding this comment.
Confirmed here: vitessio#6277
And here: https://slack-pde.slack.com/archives/C01P84R7L02/p1694468217811639
timvaillancourt
approved these changes
Sep 12, 2023
timvaillancourt
pushed a commit
that referenced
this pull request
May 27, 2024
4 tasks
timvaillancourt
added a commit
that referenced
this pull request
May 28, 2024
* add vtgate flag that explicitly allows vstream copy (#125) * fix fs.BoolVar Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * VSCopy: Resume the copy phase consistently from given GTID and lastpk (vitessio#11103) * VSCopy: Demonstrate to fail a test case on which the vstream API is supposed to resume the copy phase consistently Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Resume the copy phase consistently from given GTID and lastpk Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Build out the unit test some more Signed-off-by: Matt Lord <mattalord@gmail.com> * Update tests for new behavior Signed-off-by: Matt Lord <mattalord@gmail.com> * Improve comments Signed-off-by: Matt Lord <mattalord@gmail.com> * Limit uvstreamer changes and update test Signed-off-by: Matt Lord <mattalord@gmail.com> * Revert uvstreamer test changes Signed-off-by: Matt Lord <mattalord@gmail.com> * Revert all uvstream changes Signed-off-by: Matt Lord <mattalord@gmail.com> * VCopy: Revert the last three commits Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VCopy: Add a new vstream type that allows picking up where we left off Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VCopy: Revert the unit test change Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VCopy: Fix the end-to-end CI test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Update logic for setting up uvstreamer based on input vgtid/tablepks. Add more catchup events to test Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Refactor logic to decide if event is to be sent. Enhance unit and e2e tests. Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Don't send events for tables which we can identify as ones we haven't started copy for Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Minor changes after self-review Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Add vstream copy resume to release notes Signed-off-by: Matt Lord <mattalord@gmail.com> * Address review comments Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> * VSCopy: Send COPY_COMPLETED events when the copy operation is done (vitessio#11740) * VSCopy: Demonstrate to fail a test case on which the vstream API sends new events showing copy completed Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Send new events when the copy operation is done Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Fix typo Signed-off-by: yoheimuta <yoheimuta@gmail.com> * Initialize new map for the 'vstream * from' vtgate sql interface. Make vtadmin web protos Signed-off-by: Rohit Nayak <rohit@planetscale.com> * VSCopy: Make TestVStreamCopyBasic fail fast to avoid the end2end timeout out Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: stop sharing the 't1' table among multiple test cases running concurrently Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor the function signature to be clearer Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor the VEvents sorter to be simpler Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: refactor to stop the sorter from including a fully copied event Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> * VSCopy: Enable to copy from all shards in either a specified keyspace or all keyspaces (vitessio#11909) * VSCopy: Demonstrate to fail a test case on which the vstream API request doesn't include keyspace and shard Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Copy from all shards in all keyspaces by specifying only an empty gtid Signed-off-by: yoheimuta <yoheimuta@gmail.com> * tests: Make TestRowCount stable regardless of the number of keyspaces Signed-off-by: yoheimuta <yoheimuta@gmail.com> * tests: Cleanup TestCreateAndDropDatabase correctly to stop TestVStreamCopyWithoutKeyspaceShard from failing when running tests together Signed-off-by: yoheimuta <yoheimuta@gmail.com> * tests: Tweak to fix a comment Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: fix the unit tests when the input vgtid with an empty gtid lacks either keyspace or shard Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Keyspace wildcard selection lines up with the table wildcard selection Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Tests the VCopy with multiple keyspaces and resharding Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Make TestVStreamCopyMultiKeyspaceReshard clearer to check if the streaming two keyspaces works even after reshard Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Return an invalid argument error if shards are unspecified and gtid is neither 'current' nor empty Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a test description about its purpose and target Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Remove duplicate literals in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Retain defaultReplicas variable in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Explain why we are setting Match to 'customer.*' in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Remove an unused VStreamFlag for the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Use sentence capitalization in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Verify that we didn't lose any events or get duplicates of the keyspace being reshareded in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Return a value instead of a pointer because there is no need to modify the value Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a comment describing what TestVStreamCopyFromAllKeyspacesAndAllShards is doing and why Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a comment describing why we expect these specific numbers of events from VStream API Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Tweak the test case name Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Make a utility function to sort COPY_COMPLETED events in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Replace the matcher with a simpler one in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Move the print debug call to the FailNow section in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Use require.NoError in new tests Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Use require instead of t.Fatalf in the test Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Apply the reviewer's suggestion to make the error message easier to read Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Add a comment noting what we're actually testing Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Correct the test comment and elaborate the special-case Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Tweak an error message and a comment Signed-off-by: yoheimuta <yoheimuta@gmail.com> * VSCopy: Adjust to a change in the signature of a test function that was introduced in the main repository Signed-off-by: yoheimuta <yoheimuta@gmail.com> --------- Signed-off-by: yoheimuta <yoheimuta@gmail.com> * attempt unit test fix Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> * update test error expected Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> --------- Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com> Signed-off-by: yoheimuta <yoheimuta@gmail.com> Signed-off-by: Matt Lord <mattalord@gmail.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: pbibra <pbibra@slack-corp.com> Co-authored-by: yohei yoshimuta <yoheimuta@gmail.com> Co-authored-by: Matt Lord <mattalord@gmail.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are planning on support vstream copy testing in dev. This flag ensures we don't accidentally allow this in production until RDONLYs are properly supported for bootstrapping. This is only temporary and will be removed from our fork once we're ready in all environments.