Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

*: reduce dumpling accessing database and information_schema usage to improve its stability #305

Merged
merged 15 commits into from
Jul 15, 2021
Merged

*: reduce dumpling accessing database and information_schema usage to improve its stability #305

merged 15 commits into from
Jul 15, 2021

Conversation

lichunzhu
Copy link
Contributor

@lichunzhu lichunzhu commented Jul 12, 2021 •
edited
Loading

What problem does this PR solve?

fix #298 and fix #296 and fix #310

What is changed and how it works?

  1. Try to dump the whole table directly if we meet some errors when we try to split tables into several chunks. cfb258a

  2. Reduce accessing SelectedField and SelectedLen info from database d3ba1e8

  3. Remove the usage of information_schema 1a0d9c4, 7f7cf58, b333bea, 088b5ba, 7dac228, bdb3c26

  4. Fix the bug that dumpling finishedRows is not accurate if we retry in chunks. ff398a4

  5. Fix the bug that dumpling dumps unlocked tables for consistency lock 7f7cf58

  6. Fix the bug of no-schema argument's wrong output 09d257d

Check List

Tests

  • Unit test
  • Integration test

Side effects

  • Increased code complexity

Related changes

  • Need to cherry-pick to the release branch

Release note

  • reduce dumpling accessing database and information_schema usage and improve dumpling's stability

@lichunzhu lichunzhu mentioned this pull request Jul 14, 2021
Closed
@lichunzhu lichunzhu added this to the v4.0.14 milestone Jul 15, 2021
lance6716
lance6716 reviewed Jul 15, 2021
View reviewed changes
Copy link
Collaborator

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will review later

v4/export/config.go Outdated
@@ -169,6 +169,13 @@ func DefaultConfig() *Config {
}
}

// Clone clones a dumpling config.
func (conf *Config) Clone() *Config {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some reference type members (like TableFilter and Tables) may get not copied, is this expected?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want to use this function for tests. I will unexport this function and add it to the test code.

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the status/LGT1 One reviewer approved (LGTM1) label Jul 15, 2021
@lichunzhu
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: c625e94

@ti-chi-bot ti-chi-bot merged commit dc97ee9 into pingcap:master Jul 15, 2021
ti-chi-bot pushed a commit to ti-chi-bot/dumpling that referenced this pull request Jul 15, 2021
@lichunzhu @ti-chi-bot
This is an automated cherry-pick of pingcap#305
1d7b3fa 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Jul 15, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #313.

ti-chi-bot pushed a commit to ti-chi-bot/dumpling that referenced this pull request Jul 15, 2021
@lichunzhu @ti-chi-bot
This is an automated cherry-pick of pingcap#305
2358009 
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this pull request Jul 15, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #314.

@ti-chi-bot ti-chi-bot mentioned this pull request Jul 15, 2021
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #315.

@lichunzhu lichunzhu deleted the fixDumplingBugs branch July 16, 2021 02:24
ti-chi-bot added a commit that referenced this pull request Jul 16, 2021
@ti-chi-bot
*: reduce dumpling accessing database and information_schema usage to…
d5e0fd0 
… improve its stability (#305) (#315)
ti-chi-bot added a commit that referenced this pull request Jul 16, 2021
@ti-chi-bot
*: reduce dumpling accessing database and information_schema usage to…
2b23ba5 
… improve its stability (#305) (#313)
This was referenced Jul 22, 2021
Closed
Closed
@lichunzhu lichunzhu mentioned this pull request Aug 2, 2021
Merged
ti-chi-bot added a commit that referenced this pull request Aug 23, 2021
@ti-chi-bot
*: reduce dumpling accessing database and information_schema usage to…
0eb0fa1 
… improve its stability (#305) (#314)
This was referenced Aug 24, 2021
Merged
Merged
Merged
@ti-chi-bot ti-chi-bot mentioned this pull request Oct 12, 2021
Merged
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu
*: reduce dumpling accessing database and information_schema usage to…
23312f6 
… improve its stability (pingcap#305)
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu
*: reduce dumpling accessing database and information_schema usage to…
45c3624 
… improve its stability (pingcap#305)
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu
*: reduce dumpling accessing database and information_schema usage to…
81ed5e8 
… improve its stability (pingcap#305)
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
@lichunzhu
*: reduce dumpling accessing database and information_schema usage to…
d9b3ce5 
… improve its stability (pingcap#305)
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
@lichunzhu @tisonkun
*: reduce dumpling accessing database and information_schema usage to…
9035d63 
… improve its stability (pingcap/dumpling#305)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@lance6716 lance6716 lance6716 approved these changes

@kennytm kennytm Awaiting requested review from kennytm

Assignees
No one assigned
Labels
needs-cherry-pick-release-4.0 needs-cherry-pick-release-5.0 needs-cherry-pick-release-5.1 size/XXL status/can-merge status/LGT1 One reviewer approved (LGTM1)
Projects
None yet
Milestone
v4.0.14
3 participants
@lichunzhu @ti-chi-bot @lance6716