Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

…vitessio#11103)

* VSCopy: Demonstrate to fail a test case on which the vstream API is supposed to resume the copy phase consistently

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Resume the copy phase consistently from given GTID and lastpk

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* Build out the unit test some more

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Update tests for new behavior

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Improve comments

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Limit uvstreamer changes and update test

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Revert uvstreamer test changes

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Revert all uvstream changes

Signed-off-by: Matt Lord <mattalord@gmail.com>

* VCopy: Revert the last three commits

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VCopy: Add a new vstream type that allows picking up where we left off

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VCopy: Revert the unit test change

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VCopy: Fix the end-to-end CI test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* Update logic for setting up uvstreamer based on input vgtid/tablepks. Add more catchup events to test

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Refactor logic to decide if event is to be sent. Enhance unit and e2e tests.

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Don't send events for tables which we can identify as ones we haven't started copy for

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Minor changes after self-review

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* Add vstream copy resume to release notes

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Address review comments

Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: yoheimuta <yoheimuta@gmail.com>
Signed-off-by: Matt Lord <mattalord@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: Matt Lord <mattalord@gmail.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>

…itessio#11740)

* VSCopy: Demonstrate to fail a test case on which the vstream API sends new events showing copy completed

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Send new events when the copy operation is done

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Fix typo

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* Initialize new map for the 'vstream * from' vtgate sql interface. Make vtadmin web protos

Signed-off-by: Rohit Nayak <rohit@planetscale.com>

* VSCopy: Make TestVStreamCopyBasic fail fast to avoid the end2end timeout out

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: stop sharing the 't1' table among multiple test cases running concurrently

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: refactor the function signature to be clearer

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: refactor the VEvents sorter to be simpler

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: refactor to stop the sorter from including a fully copied event

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

Signed-off-by: yoheimuta <yoheimuta@gmail.com>
Signed-off-by: Rohit Nayak <rohit@planetscale.com>
Co-authored-by: Rohit Nayak <rohit@planetscale.com>

… or all keyspaces (vitessio#11909)

* VSCopy: Demonstrate to fail a test case on which the vstream API request doesn't include keyspace and shard

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Copy from all shards in all keyspaces by specifying only an empty gtid

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* tests: Make TestRowCount stable regardless of the number of keyspaces

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* tests: Cleanup TestCreateAndDropDatabase correctly to stop TestVStreamCopyWithoutKeyspaceShard from failing when running tests together

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* tests: Tweak to fix a comment

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: fix the unit tests when the input vgtid with an empty gtid lacks either keyspace or shard

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Keyspace wildcard selection lines up with the table wildcard selection

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Tests the VCopy with multiple keyspaces and resharding

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Make TestVStreamCopyMultiKeyspaceReshard clearer to check if the streaming two keyspaces works even after reshard

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Return an invalid argument error if shards are unspecified and gtid is neither 'current' nor empty

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Add a test description about its purpose and target

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Remove duplicate literals in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Retain defaultReplicas variable in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Explain why we are setting Match to 'customer.*' in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Remove an unused VStreamFlag for the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Use sentence capitalization in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Verify that we didn't lose any events or get duplicates of the keyspace being reshareded in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Return a value instead of a pointer because there is no need to modify the value

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Add a comment describing what TestVStreamCopyFromAllKeyspacesAndAllShards is doing and why

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Add a comment describing why we expect these specific numbers of events from VStream API

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Tweak the test case name

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Make a utility function to sort COPY_COMPLETED events in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Replace the matcher with a simpler one in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Move the print debug call to the FailNow section in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Use require.NoError in new tests

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Use require instead of t.Fatalf in the test

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Apply the reviewer's suggestion to make the error message easier to read

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Add a comment noting what we're actually testing

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Correct the test comment and elaborate the special-case

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Tweak an error message and a comment

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

* VSCopy: Adjust to a change in the signature of a test function that was introduced in the main repository

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

---------

Signed-off-by: yoheimuta <yoheimuta@gmail.com>

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>

Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>