Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

mydumper: generate tables needed to dump for mydumper automatically #310

Merged
merged 23 commits into from
Oct 18, 2019
Merged

mydumper: generate tables needed to dump for mydumper automatically #310

merged 23 commits into from
Oct 18, 2019

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Oct 14, 2019 •
edited
Loading

What problem does this PR solve?

Now users need to add extra-args for mydumper to help it decide tables needed to dump. When the situations are very complicated it's very hard for users to do that. This PR aims to generate the tables needed to be dumped automatically so that we can simplify the configuration process.

What is changed and how it works?

Generate mydumper extra-args through black-white list and router rules when it is empty. Before start mydumper will connect to DB to fetch all the tables from database, filter useful tables and generate extra-args in --tables-list db1.tb1,db2.tb2.. format.
A unit-test is added to test whether it can generate right args when extra-args is empty. For integration_tests except "start_task" I expurge the extra-args to test whether it can run correctly.

Check List

Tests

  • Unit test
  • Integration test

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to be included in the release note

@lichunzhu lichunzhu added priority/normal Minor change, requires approval from ≥1 primary reviewer needs-update-docs Should update docs after this PR is merged. Remove this label once the docs are updated status/PTAL This PR is ready for review. Add this label back after committing new changes needs-cherry-pick-release-1.0 This PR should be cherry-picked to release-1.0. Remove this label after cherry-picked to release-1.0 needs-update-release-note This PR should be added into release notes. Remove this label once the release notes are updated labels Oct 14, 2019
@CLAassistant
Copy link

CLAassistant commented Oct 14, 2019 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@lichunzhu lichunzhu added status/WIP This PR is still work in progress and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Oct 14, 2019
@codecov
Copy link

codecov bot commented Oct 14, 2019 •
edited
Loading

Codecov Report

❗ No coverage uploaded for pull request base (master@26b90b3). Click here to learn what that means.
The diff coverage is 70.2702%.

@@             Coverage Diff             @@
##             master       #310   +/-   ##
===========================================
  Coverage          ?   59.9082%           
===========================================
  Files             ?        135           
  Lines             ?      15043           
  Branches          ?          0           
===========================================
  Hits              ?       9012           
  Misses            ?       5155           
  Partials          ?        876

@lichunzhu lichunzhu added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/WIP This PR is still work in progress labels Oct 14, 2019
WangXiangUSTC
WangXiangUSTC reviewed Oct 14, 2019
View reviewed changes
mydumper/util.go Outdated Show resolved Hide resolved
mydumper/util.go Outdated Show resolved Hide resolved
mydumper/mydumper.go Outdated Show resolved Hide resolved
@lichunzhu lichunzhu added status/WIP This PR is still work in progress and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Oct 14, 2019
@lichunzhu
fix bugs, change to strings.Join and add extraArgs justifications to …
23c71b6 
…check whether we should generate tables args or not
@lichunzhu lichunzhu added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/WIP This PR is still work in progress labels Oct 14, 2019
@lichunzhu lichunzhu closed this Oct 14, 2019
@lichunzhu lichunzhu reopened this Oct 14, 2019
@lichunzhu
Copy link
Contributor Author

@WangXiangUSTC Revised, PTAL.

WangXiangUSTC
WangXiangUSTC reviewed Oct 15, 2019
View reviewed changes
Copy link
Contributor

@WangXiangUSTC WangXiangUSTC left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

dm/worker/worker_test.go Show resolved Hide resolved
mydumper/mydumper_test.go Outdated Show resolved Hide resolved
@WangXiangUSTC
Copy link
Contributor

@csuzhangxc PTAL

csuzhangxc
csuzhangxc reviewed Oct 17, 2019
View reviewed changes
Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

tests/sharding/conf/dm-task.yaml Show resolved Hide resolved
pkg/baseconn/conn.go Outdated Show resolved Hide resolved
pkg/utils/common.go Outdated Show resolved Hide resolved
mydumper/util.go Show resolved Hide resolved
mydumper/mydumper.go Show resolved Hide resolved
mydumper/mydumper_test.go Outdated Show resolved Hide resolved
csuzhangxc
csuzhangxc reviewed Oct 17, 2019
View reviewed changes
mydumper/util.go Outdated Show resolved Hide resolved
csuzhangxc
csuzhangxc reviewed Oct 18, 2019
View reviewed changes
pkg/utils/common.go Outdated Show resolved Hide resolved
csuzhangxc
csuzhangxc reviewed Oct 18, 2019
View reviewed changes
Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

mydumper/mydumper.go Outdated Show resolved Hide resolved
csuzhangxc
csuzhangxc approved these changes Oct 18, 2019
View reviewed changes
Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csuzhangxc csuzhangxc added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Oct 18, 2019
@lichunzhu lichunzhu merged commit e512948 into pingcap:master Oct 18, 2019
@lichunzhu lichunzhu deleted the czli/mydumper/configSimplify branch October 18, 2019 10:44
@sre-bot sre-bot mentioned this pull request Oct 18, 2019
Merged
@sre-bot
Copy link

sre-bot commented Oct 18, 2019

cherry pick to release-1.0 in PR #326

@sre-bot sre-bot added already-cherry-pick-1.0 The related PR is already cherry-picked to release-1.0. Add this label once the PR is cherry-picked and removed needs-cherry-pick-release-1.0 This PR should be cherry-picked to release-1.0. Remove this label after cherry-picked to release-1.0 labels Oct 18, 2019
lichunzhu pushed a commit that referenced this pull request Oct 21, 2019
@sre-bot @lichunzhu
mydumper: generate tables needed to dump for mydumper automatically (#…
61ad0ca 
…310) (#326)

generate mydumper extra-args automatically from DB through black-white list and router rules when it is empty.
@lichunzhu lichunzhu mentioned this pull request Oct 31, 2019
Merged
@csuzhangxc csuzhangxc added already-update-release-note The release note is updated. Add this label once the release note is updated and removed needs-update-release-note This PR should be added into release notes. Remove this label once the release notes are updated labels Nov 1, 2019
@csuzhangxc csuzhangxc added already-update-docs The docs related to this PR already updated. Add this label once the docs are updated and removed needs-update-docs Should update docs after this PR is merged. Remove this label once the docs are updated labels Mar 13, 2020
lichunzhu added a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020
@lichunzhu
mydumper: generate tables needed to dump for mydumper automatically (p…
09085dd 
…ingcap#310)

generate the tables needed to be dumped for mydumper automatically when tables needed to be dumped are not given.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@WangXiangUSTC WangXiangUSTC WangXiangUSTC left review comments

@csuzhangxc csuzhangxc csuzhangxc approved these changes

Assignees
No one assigned
Labels
already-cherry-pick-1.0 The related PR is already cherry-picked to release-1.0. Add this label once the PR is cherry-picked already-update-docs The docs related to this PR already updated. Add this label once the docs are updated already-update-release-note The release note is updated. Add this label once the release note is updated priority/normal Minor change, requires approval from ≥1 primary reviewer priority/release-blocker This PR blocks a release. Please review it ASAP. status/LGT2 Two reviewers already commented LGTM, ready for merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@lichunzhu @CLAassistant @WangXiangUSTC @sre-bot @csuzhangxc