vrepl: Prep for lookup vindex backfill #5669
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>
systay
pushed a commit
that referenced
this pull request
Jul 22, 2024
…5669) * Vectors: Online DDL support Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Add support for vectors in vreplication workflows. unit and e2e tests. Update cnf file to add loose_with_ps_vectors Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Self review fixes. Add e2e TestVector to CI Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Simplify unit test CI workflow to debug failure Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Skip test on errors on setting with_ps_vectors Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Revert debug mods to unit_race.yml Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Unit test race is again failing with no logs. So modify CI workflow temporarily to get full detail to debug random failures apparently unrelated to this PR Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Setup templating for vectors vreplication build Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Add vcopier unit test. Insert additional rows in e2e test after copy phase to test replication phase Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Switch shard for vector test to correct one Signed-off-by: Rohit Nayak <rohit@planetscale.com> * Use vectors build for unit tests Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Refactor tests: move vector unit tests to separate files Signed-off-by: Rohit Nayak <rohit@planetscale.com> * endtoend suite: filter with ONLINEDDL_SUITE_TEST_FILTER Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * recognize ERVectorDimensionsMismatch = ErrorCode(4501) as unrecoverable Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add expected failign vector test. Rename vector tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * recognize ERVectorDimensionsMismatch = ErrorCode(4501) as unrecoverable Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Unit race still failing without details:-(. Commenting out the summary code which seems to be redacting the details Signed-off-by: Rohit Nayak <rohit@planetscale.com> * adding onlineddl_vector_stress CI tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * validate row count primary vs replica Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * comparing primary and replica data Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * onlineddl_vector_stress in clustersRequiringVectors Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Adding workflow file Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix test assertion/concurrency; accept specific errors Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more queries in transaction Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * remove testing.T from goroutines Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * uncomment static/workload tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * 'onlineddl_vrepl_suite' now requires a vectors build Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * explicit primary/replica test names Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * use vector build Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update mysql version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * apply mysql version to vector stress workflow Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * updating MySQL image version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fixes per review Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update mysql image version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update mysql image version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * bail out early if the migration si failed. Accept error 1317 (ERQueryInterrupted) as valid Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more DDL scenarios Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more scenario Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * using --force-cut-over-after=30s Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * support direct alter. Support ONLINEDDL_VECTOR_TEST_FILTER environment variable. More tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add test: dropping all indexes Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add test: adding a null index (index on a column that only has nulls) Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * one vector column is not null Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * adding mixed direct & onlineddl test scenario Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * preWorkloadQuery, preCutoverQuery, more tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more test scenario Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * fix sql error Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update MySQL image Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * add massDML option to run table-wide UPDATE or DELETE Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * remove debugging sections Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * make generate_ci_workflows Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * more test scenarios Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * increase test timeout Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * introducing 'repeats' Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * improved filtering Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Adding shards for onlineddl_fulltext_stress, onlineddl_blob_stress. Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * wording Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * LIMIT 10000 for when we increase number of rows Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update go version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Vectors: update to MySQL 9.0 syntax Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update MySQL image version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update to mysql 8.0 vector function names Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * resolved conflict Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update mysql image version for unit tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update mysql image version for unit tests Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * allow changing vector length Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * adapt new column_length and decimals Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Vectors: add vector column and data in 'backup_pitr' Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update mysql image version Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * resolve conflicts Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * resolve conflicts Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Add 'backup_pitr' to list of vector-enabled tests. This help reducing conflicts with planetscale/vitess-private#5659 Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * remove blob_stress and fulltext_stress as they were only introduced to test START_GTID inconsistencies. This has since evolved to planetscale/vitess-private#5629 and external to this PR Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * Update evalengine to handle basic vector types Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Add vector dimensionality medetata flag Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Remove custom vector check This now should have it's own specific type in MySQL. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * update mysql image Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * update vector function Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * backup tests: distinguish test with vector support and without vector support Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> * remove onlineddl_fulltext_stress and onlineddl_blob_stress Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> --------- Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com> Signed-off-by: Rohit Nayak <rohit@planetscale.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Co-authored-by: Rohit Nayak <rohit@planetscale.com> Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To prep for the lookup vindex backfill feature, we need to add support for the
write_onlyoption to consistent lookup vindexes.Additionally, we also need a vreplication option to stop after the backfill is done:
stop_after_copy.