Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VTShovel - VReplication support for external databases #5289

Merged
merged 33 commits into from
Dec 5, 2019
Merged

VTShovel - VReplication support for external databases #5289

merged 33 commits into from
Dec 5, 2019

Commits on Oct 3, 2019

  1. Adds proto definition to store dml extracted from Query 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 3, 2019
    Configuration menu
    Copy the full SHA
    489eb3c View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2019

  1. Update schema engine to expect mysql.ConnParams 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    aea60ab View commit details
    Browse the repository at this point in the history

  2. Adds support for file:pos in mysql binlogdump interface 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    11301e3 View commit details
    Browse the repository at this point in the history

  3. Adds BinlogFilePos as a way to encode mysql replication 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    2f1d3b1 View commit details
    Browse the repository at this point in the history

  4. Adds StartBinlogDumpFromFilePosition to slave connection protocol 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    a2b0074 View commit details
    Browse the repository at this point in the history

  5. Enables vreplication to run directly from MySQL 

    * Adds support for VStream to start from filename:pos and not gtid sets.
* Adds support for statement based replication streams (this should only be used
  in the context of mysql streamer, it is not safe for tablet vreplicaiton).
* Adds support to run vstream from mysql directly

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    4210649 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'upstream-master' into vtshovel-poc

    @rafael
    rafael committed Oct 10, 2019
    Configuration menu
    Copy the full SHA
    f903605 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2019

  1. Adds the core of vtshovel program 

    * Adds binary to run vtshovel.
* At the moment only working in ephemeral mode (i.e no data is persisted back to
  vrsettings).
* vtshovel only works for statement based replication right now. This is due to
  now having a good way to have a schema loader. We will itereate on this.

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 11, 2019
    Configuration menu
    Copy the full SHA
    c3c238b View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2019

  1. Simplifies vtshovel logic. It assumes that it runs directly again 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 16, 2019
    Configuration menu
    Copy the full SHA
    8bf38cc View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2019

  1. Update approach to not require another binary to run vtshovel 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 17, 2019
    1 Configuration menu
    Copy the full SHA
    b56bf67 View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2019

  1. Fixes some bugs in dbconfigs and vstreamer client after inital testing 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 18, 2019
    Configuration menu
    Copy the full SHA
    c7926ef View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2019

  1. Adds QPS chart to tablet vreplication section 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 21, 2019
    Configuration menu
    Copy the full SHA
    46c0fe4 View commit details
    Browse the repository at this point in the history

  2. Adds flag to register metrics. 

    * This will be removed in future PR. Adding while in POC

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 21, 2019
    Configuration menu
    Copy the full SHA
    e73faef View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2019

  1. Addresses comments from review 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 29, 2019
    Configuration menu
    Copy the full SHA
    cf57589 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2019

  1. WIP: Adds test for vstreamer client 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 30, 2019
    Configuration menu
    Copy the full SHA
    90ee27a View commit details
    Browse the repository at this point in the history

  2. Fixes bug in dbconfigs that was causing vstreamer to not work correctly 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Oct 30, 2019
    Configuration menu
    Copy the full SHA
    3fc6b08 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2019

  1. Adds tests for vstreamer_client 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    b674d5e View commit details
    Browse the repository at this point in the history

  2. Do not pass source conn params around 

    * At the moment we only support erpel user. Passing source conn params around
was adding unnecessary complexity.
* This cleans up that and makes it more explicit that only erepl user is
supported. In the future we will add more flexibility in terms of what kind of
users can be configured for external vreplication streams

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    9988496 View commit details
    Browse the repository at this point in the history

  3. Style improvements 

    * Fix typo in some comments.
* Make VReplicator private again. This change is no longer needed. Originally we
wanted "vtshovel" to be an external process. Given that this now hooks into the
existent engine, there is no need to make this public.

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 6, 2019
    Configuration menu
    Copy the full SHA
    4bd4904 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2019

  1. Abort on error when executing a DML in statement based replication 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 14, 2019
    Configuration menu
    Copy the full SHA
    21eeeb4 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2019

  1. Merge branch 'upstream-master' into vtshovel-poc

    @rafael
    rafael committed Nov 23, 2019
    Configuration menu
    Copy the full SHA
    6593e5d View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2019

  1. Fixes per rebase with file:pos feature 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 26, 2019
    Configuration menu
    Copy the full SHA
    1665701 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/vitessio/vitess into vtsh… 

    …ovel-poc
    @rafael
    rafael committed Nov 26, 2019
    Configuration menu
    Copy the full SHA
    67ac881 View commit details
    Browse the repository at this point in the history

  3. Revert no longer needed changes in vstream 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 26, 2019
    Configuration menu
    Copy the full SHA
    731f02f View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2019

  1. Adds support to set flavor for specific connections 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 27, 2019
    Configuration menu
    Copy the full SHA
    ec632f2 View commit details
    Browse the repository at this point in the history

  2. Fixes per integration with file:pos rebase 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 27, 2019
    Configuration menu
    Copy the full SHA
    d4460ee View commit details
    Browse the repository at this point in the history

  3. Remove test to make sure this is the last outstanding issue 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 27, 2019
    Configuration menu
    Copy the full SHA
    4f03489 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2019

  1. Fixes bug in filepos and adds test for statement mode 

    * StripChecksum was changing the type of the event. This was a bug.
* Adds test to vstreamer to reflect new support for statement based replication

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 28, 2019
    Configuration menu
    Copy the full SHA
    b1a8772 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/vitessio/vitess into vtsh… 

    …ovel-poc
    @rafael
    rafael committed Nov 28, 2019
    Configuration menu
    Copy the full SHA
    69a4dd0 View commit details
    Browse the repository at this point in the history

  3. Fixes per changes upstream 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Nov 28, 2019
    Configuration menu
    Copy the full SHA
    59785b6 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2019

  1. Adds more tests and fixes govet issues 

    Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Dec 1, 2019
    Configuration menu
    Copy the full SHA
    5fd8925 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2019

  1. Re-running go-imports and addressing other comments from PR 

    * Compute canAcceptStmtEvents when creating vplayer.

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
    @rafael
    rafael committed Dec 4, 2019
    Configuration menu
    Copy the full SHA
    559a210 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/vitessio/vitess into vtsh… 

    …ovel-poc
    @rafael
    rafael committed Dec 4, 2019
    Configuration menu
    Copy the full SHA
    dc0f4a3 View commit details
    Browse the repository at this point in the history