Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

syncer: use downstream PK/UK to generate DML #2163

Closed
wants to merge 29 commits into from

Conversation

WizardXiao
Copy link

@WizardXiao WizardXiao commented Sep 23, 2021

What problem does this PR solve?

#1895

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Code changes

  • Has exported function/method change
  • Has exported variable/fields change
  • Has interface methods change
  • Has persistent data change

Side effects

  • Possible performance regression
  • Increased code complexity
  • Breaking backward compatibility

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to update the dm/dm-ansible
  • Need to be included in the release note

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@CLAassistant
Copy link

CLAassistant commented Sep 23, 2021

CLA assistant check
All committers have signed the CLA.

@ti-chi-bot
Copy link
Member

Welcome @WizardXiao!

It looks like this is your first PR to pingcap/dm 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/dm. 😃

@Ehco1996 Ehco1996 changed the title commit-message: update the schema tracker core code about #1895 schema-tracker: update the schema tracker core code about #1895 Sep 23, 2021
@Ehco1996 Ehco1996 changed the title schema-tracker: update the schema tracker core code about #1895 syncer: use downstream PK/UK to generate DML Sep 23, 2021
@Ehco1996
Copy link
Contributor

@WizardXiao Thank you for your contribution 🎉🎉 is this PR ready for review?

a few more suggestions:

  • is there a publicly available spec design document? If so, please post it in the description of the PR
  • please fix the errors in the CI to ensure that it passes.

@pingcap pingcap deleted a comment from WizardXiao Sep 24, 2021
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
syncer/syncer_test.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
@okJiang okJiang dismissed their stale review September 27, 2021 02:29

Misoperation

Copy link
Contributor

@lichunzhu lichunzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will review later

pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
syncer/dml.go Outdated Show resolved Hide resolved
syncer/dml.go Outdated Show resolved Hide resolved
syncer/dml.go Outdated Show resolved Hide resolved
@lance6716
Copy link
Collaborator

/run-unit-tests

@Ehco1996
Copy link
Contributor

some ut failed and panic

----------------------------------------------------------------------
FAIL: syncer_test.go:1222: testSyncerSuite.TestExitSafeModeByConfig

syncer_test.go:1370:
    checkJobs(c, testJobs.jobs, expectJobs)
syncer_test.go:1604:
    c.Assert(len(jobs), Equals, len(expectJobs), Commentf("jobs = %q", jobs))
... obtained int = 6
... expected int = 12
... jobs = ["tp: flush, sql: , args: [], key: , ddls: [], last_location: position: (, 0), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 0), gtid-set: " "tp: ddl, sql: , args: [], key: , ddls: [CREATE DATABASE IF NOT EXISTS `test_1`], last_location: position: (, 1938), gtid-set: , start_location: position: (, 1871), gtid-set: , current_location: position: (, 1938), gtid-set: " "tp: flush, sql: , args: [], key: , ddls: [], last_location: position: (, 0), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 0), gtid-set: " "tp: ddl, sql: , args: [], key: , ddls: [CREATE TABLE IF NOT EXISTS `test_1`.`t_1` (`id` INT PRIMARY KEY,`name` VARCHAR(24))], last_location: position: (, 2107), gtid-set: , start_location: position: (, 2003), gtid-set: , current_location: position: (, 2107), gtid-set: " "tp: insert, sql: REPLACE INTO `test_1`.`t_1` (`id`,`name`) VALUES (?,?), args: [1 a], key: , ddls: [], last_location: position: (, 2214), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 2307), gtid-set: " "tp: flush, sql: , args: [], key: , ddls: [], last_location: position: (, 0), gtid-set: , start_location: position: (, 0), gtid-set: , current_location: position: (, 0), gtid-set: "]


----------------------------------------------------------------------
PANIC: syncer_test.go:714: testSyncerSuite.TestGeneratedColumn

... Panic: runtime error: invalid memory address or nil pointer dereference (PC=0x1415466)

/usr/local/go/src/runtime/panic.go:971
  in gopanic
/usr/local/go/src/runtime/panic.go:212
  in panicmem
/usr/local/go/src/runtime/signal_unix.go:734
  in sigpanic
/usr/local/go/src/database/sql/sql.go:1197
  in DB.conn
/usr/local/go/src/database/sql/sql.go:1553
  in DB.exec
/usr/local/go/src/database/sql/sql.go:1535
  in DB.ExecContext
/usr/local/go/src/database/sql/sql.go:1549
  in DB.Exec
syncer_test.go:716
  in testSyncerSuite.TestGeneratedColumn
/usr/local/go/src/reflect/value.go:337
  in Value.Call
/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:850
  in suiteRunner.forkTest.func1
/nfs/cache/mod/github.com/pingcap/check@v0.0.0-20200212061837-5e12011dc712/check.go:739
  in suiteRunner.forkCall.func1
/usr/local/go/src/runtime/asm_amd64.s:1371
  in goexit

pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
@WizardXiao
Copy link
Author

/run-unit-test

Copy link
Collaborator

@lance6716 lance6716 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will review later

pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Outdated Show resolved Hide resolved
tests/downstream_diff_index/run.sh Show resolved Hide resolved
tests/downstream_diff_index/run.sh Show resolved Hide resolved
pkg/schema/tracker_test.go Show resolved Hide resolved
pkg/schema/tracker_test.go Show resolved Hide resolved
pkg/schema/tracker_test.go Show resolved Hide resolved
Copy link
Contributor

@Ehco1996 Ehco1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this work is almost done 🎉
only some light changes need to be applied after your merge master code, remeber to update the genSql logic ( prepare index/table info before gensql in case of accessing tracker concurrently )

btw, please reslove some outdated comment for better reveiw 😉

pkg/schema/tracker.go Outdated Show resolved Hide resolved
pkg/schema/tracker.go Show resolved Hide resolved
@WizardXiao
Copy link
Author

i think this work is almost done 🎉 only some light changes need to be applied after your merge master code, remeber to update the genSql logic ( prepare index/table info before gensql in case of accessing tracker concurrently )

btw, please reslove some outdated comment for better reveiw 😉

thanks, i have merged master code and fixed unit-test/integration-test.

@lance6716
Copy link
Collaborator

/cc @Ehco1996

@ti-chi-bot ti-chi-bot requested a review from Ehco1996 October 29, 2021 06:25
@lance6716
Copy link
Collaborator

/run-dm-integration-tests

@WizardXiao
Copy link
Author

This PR has been moved to https://github.com/pingcap/ticdc/pull/3168

@WizardXiao WizardXiao closed this Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
contribution For contributor first-time-contributor needs-cherry-pick-release-2.0 This PR should be cherry-picked to release-2.0. Remove this label after cherry-picked to release-2.0 size/XXL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants