Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VReplication: Initialize Sequence Tables Used By Tables Being Moved #13656

Merged
merged 62 commits into from
Aug 9, 2023
Merged

VReplication: Initialize Sequence Tables Used By Tables Being Moved #13656

merged 62 commits into from
Aug 9, 2023

Commits on Jul 28, 2023

  1. Get initial PoC implementation working for initializing sequence 

    tables being used by tables being moved

Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    5d5e6d7 View commit details
    Browse the repository at this point in the history

  2. Reset cache per table, only when needed 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    31d3895 View commit details
    Browse the repository at this point in the history

  3. Gather sequence details after keyspace lock but before blocking writes 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    ed62652 View commit details
    Browse the repository at this point in the history

  4. Check for context cancellation often 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    5ac2e4d View commit details
    Browse the repository at this point in the history

  5. Skip sequence work for sharded to sharded migrations 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    52ab8b7 View commit details
    Browse the repository at this point in the history

  6. Efficiency improvements 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    f2b7313 View commit details
    Browse the repository at this point in the history

  7. More tweaks 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    ca06b32 View commit details
    Browse the repository at this point in the history

  8. Address todo 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    91112ae View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2023

  1. Search keyspaces in parallel 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 29, 2023
    Configuration menu
    Copy the full SHA
    0c3e0f6 View commit details
    Browse the repository at this point in the history

  2. Parallelize init work 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 29, 2023
    Configuration menu
    Copy the full SHA
    fcab612 View commit details
    Browse the repository at this point in the history

  3. Consolidate selects 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 29, 2023
    Configuration menu
    Copy the full SHA
    b4037da View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2023

  1. Minor tweaks after self review 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    f1c95b7 View commit details
    Browse the repository at this point in the history

  2. Add flag and switcher ifc impl 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    fecfc49 View commit details
    Browse the repository at this point in the history

  3. Add flag and vtctldclient impl 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    72a5c21 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into vrepl_seq_init 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    4ccd341 View commit details
    Browse the repository at this point in the history

  5. Add missing returns 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 30, 2023
    Configuration menu
    Copy the full SHA
    142109a View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. Concurrency improvements 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    e92b498 View commit details
    Browse the repository at this point in the history

  2. Update wrangler unit tests 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    65c17d1 View commit details
    Browse the repository at this point in the history

  3. Enable new flag in e2e test 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    cacf7d9 View commit details
    Browse the repository at this point in the history

  4. Deflake SwitchTraffic dry run unit tests 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    e352088 View commit details
    Browse the repository at this point in the history

  5. Tweakin' and deflakin' 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Jul 31, 2023
    Configuration menu
    Copy the full SHA
    bee2a67 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Add e2e test integration 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    252b791 View commit details
    Browse the repository at this point in the history

  2. Tweaks and bug fixes 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    36f9059 View commit details
    Browse the repository at this point in the history

  3. Fix vtctldclient bug and enable usage for ReverseTraffic 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    17227ff View commit details
    Browse the repository at this point in the history

  4. Adjust VDiff2 e2e tests 

    The rows inserted before resume need to have a PK greater
than the rows inserted in the initial customer schema, which
is now high in order to test the target sequence initialization.

Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 1, 2023
    Configuration menu
    Copy the full SHA
    026bc4b View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. Move keyspace search concurrency to errgroup 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    dbf7d84 View commit details
    Browse the repository at this point in the history

  2. Move init function to errgroup 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    c40e6cc View commit details
    Browse the repository at this point in the history

  3. Move vtctldclient impl to errgroup 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    c0e5b97 View commit details
    Browse the repository at this point in the history

  4. Fix duh bug 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    331d60f View commit details
    Browse the repository at this point in the history

  5. Use slices.Sort in dry run and remove DEBUG logs 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    f52ac01 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/main' into vrepl_seq_init 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    7ac0d7a View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. WiP unit test work 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    4f428d7 View commit details
    Browse the repository at this point in the history

  2. Address some review comments 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    4226af4 View commit details
    Browse the repository at this point in the history

  3. Address unit test race 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    2e5822a View commit details
    Browse the repository at this point in the history

  4. Finish unit test todos 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    2fed344 View commit details
    Browse the repository at this point in the history

  5. use correct ctx 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    8a77582 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/main' into vrepl_seq_init 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 3, 2023
    Configuration menu
    Copy the full SHA
    bae88b8 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. Implement a comment suggestion 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    28af2ee View commit details
    Browse the repository at this point in the history

  2. Make same switch to Cut in wrangler impl 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    2ca52b2 View commit details
    Browse the repository at this point in the history

  3. Align vtctlclient help output with reality 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    eb667fc View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/main' into vrepl_seq_init 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    84a0bdf View commit details
    Browse the repository at this point in the history

  5. Fix after merging main 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    a82d853 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2023

  1. Address another review comment 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 5, 2023
    Configuration menu
    Copy the full SHA
    a17bef6 View commit details
    Browse the repository at this point in the history

  2. Set the backing table's default DB name when fully qualified 

    Otherwise if they're all fully qualified we would not have
it set as it was only being done during the search.

Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 5, 2023
    Configuration menu
    Copy the full SHA
    29f978d View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2023

  1. Minor tweaks after self review of recent changes 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 6, 2023
    Configuration menu
    Copy the full SHA
    71ff024 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2023

  1. Merge remote-tracking branch 'origin/main' into vrepl_seq_init 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    1898433 View commit details
    Browse the repository at this point in the history

  2. Deflake tests that use tabletconntest.SetProtocol() 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    22e21d0 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into vrepl_seq_init 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    341b8be View commit details
    Browse the repository at this point in the history

  4. Register grpc dialer for healthcheck test 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    4a1f2eb View commit details
    Browse the repository at this point in the history

  5. More deflaking 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    4b53f66 View commit details
    Browse the repository at this point in the history

  6. Add another bounds safety check 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    a04b965 View commit details
    Browse the repository at this point in the history

  7. Try to deflake the vtctldclient MoveTables unit test 

    I have only seen this fail in the CI's MySQL57 unit test workflow.

The order that the set foreign_key_checks=1 statement comes in is
for some reason non-deterministic in that workflow specifically.

Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 8, 2023
    Configuration menu
    Copy the full SHA
    c00925c View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. Improve and unify error handling when switching reads/writes 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    89295b5 View commit details
    Browse the repository at this point in the history

  2. Address review comments 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    9600566 View commit details
    Browse the repository at this point in the history

  3. Use closure to more safely manage mutex 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    fb01b93 View commit details
    Browse the repository at this point in the history

  4. Forgot to remove the unlock... 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    0663f6f View commit details
    Browse the repository at this point in the history

  5. Improve error msg and update unit test expectations 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    c147748 View commit details
    Browse the repository at this point in the history

  6. Tweaks for unit tests 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    a875009 View commit details
    Browse the repository at this point in the history

  7. We don't need the explicit ignore any more. 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    7de744f View commit details
    Browse the repository at this point in the history

  8. Attempts to do more deflaking 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    9202a4c View commit details
    Browse the repository at this point in the history

  9. Squeeze in another vtctldclient fix 

    This was just noticed when helping with an issue.

Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    7f5d0a1 View commit details
    Browse the repository at this point in the history

  10. Building on the last commit to cleanup option handling 

    Signed-off-by: Matt Lord <mattalord@gmail.com>
    @mattlord
    mattlord committed Aug 9, 2023
    Configuration menu
    Copy the full SHA
    4d6d05c View commit details
    Browse the repository at this point in the history