Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

*: support generated column in loader and binlog replication #42

Merged
merged 20 commits into from
Feb 21, 2019
Merged

*: support generated column in loader and binlog replication #42

merged 20 commits into from
Feb 21, 2019

Conversation

amyangfei
Copy link
Contributor

What problem does this PR solve?

support synchronization data with generated column

What is changed and how it works?

  1. auto filter generated column when generating DDL or DML
  2. auto add column name list when generating insert SQL with generated column in loader

Check List

Tests

  • Unit test
  • Integration test

need to upgrade test MySQL to 5.7

Side effects

  • Increased code complexity

Related changes

  • Need to be included in the release note

@amyangfei amyangfei added priority/normal Minor change, requires approval from ≥1 primary reviewer status/WIP This PR is still work in progress type/bug-fix Bug fix type/qa relate to quality assurance labels Feb 12, 2019
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei amyangfei added status/PTAL This PR is ready for review. Add this label back after committing new changes and removed status/WIP This PR is still work in progress labels Feb 13, 2019
@amyangfei
Copy link
Contributor Author

PTAL @GregoryIan @csuzhangxc

@amyangfei
Copy link
Contributor Author

/run-all-tests

3 similar comments
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

@IANTHEREAL
Copy link
Collaborator

deal with the ci coverage decline @amyangfei

@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

deal with the ci coverage decline @amyangfei

I find the instability of coverage rate relates to some cleanup staff, for example we can compare difference between https://coveralls.io/builds/21662660/source?filename=syncer/db.go#L280 and https://coveralls.io/builds/21662523/source?filename=syncer/db.go#L280 .
All new code in this commit are tested in integration test or unit test. I will make another pr to try to make coverage more stable

@amyangfei
Copy link
Contributor Author

/run-all-tests

2 similar comments
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

csuzhangxc
csuzhangxc reviewed Feb 18, 2019
View reviewed changes
loader/convert_data.go Outdated Show resolved Hide resolved
IANTHEREAL
IANTHEREAL reviewed Feb 18, 2019
View reviewed changes
loader/convert_data_test.go Outdated
},
}

for i, r := range rules {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rules only contains one rule，but expected contains two case

csuzhangxc
csuzhangxc reviewed Feb 18, 2019
View reviewed changes
syncer/dml.go Outdated Show resolved Hide resolved
syncer/dml.go Show resolved Hide resolved
csuzhangxc
csuzhangxc reviewed Feb 18, 2019
View reviewed changes
tests/sharding/data/db1.increment.sql
insert into t1 (uid, name, info) values (10005, 'Buenos Aires', '{"age": 100}');
insert into t2 (uid, name, info) values (20007, 'Buenos Aires', '{"age": 200}');
alter table t1 add key multi_col_idx(uid, id_gen);
alter table t2 add key multi_col_idx(uid, id_gen);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can it work if no primary key, but only multi_col_idx including generated column?

@amyangfei
Copy link
Contributor Author

/run-all-tests

1 similar comment
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

PTAL @GregoryIan @csuzhangxc

csuzhangxc
csuzhangxc reviewed Feb 20, 2019
View reviewed changes
Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

syncer/dml.go Outdated Show resolved Hide resolved
@amyangfei
Copy link
Contributor Author

/run-all-tests

3 similar comments
@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

@amyangfei
Copy link
Contributor Author

/run-all-tests

IANTHEREAL
IANTHEREAL reviewed Feb 21, 2019
View reviewed changes
syncer/db.go
@@ -456,3 +459,7 @@ func getBinaryLogs(db *sql.DB) ([]binlogSize, error) {
}
return files, nil
}

func (c *column) isGeneratedColumn() bool {
return strings.Contains(c.extra, "VIRTUAL GENERATED") || strings.Contains(c.extra, "STORED GENERATED")
Copy link
Collaborator

@IANTHEREAL IANTHEREAL Feb 21, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must extra be uppercase? should we use == strings.TrimSpace(xxx) rather than strings.Contains?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to MySQL doc, VIRTUAL GENERATED and STORED GENERATED in Extra field are always uppercase

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strings.Contains is used because there may exist other data in Extra?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid some comment options are in it

Copy link
Contributor Author

@amyangfei amyangfei Feb 21, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://dev.mysql.com/doc/refman/5.7/en/show-columns.html
In MySQL doc, there are only four Extra fields: auto_increment, on update CURRENT_TIMESTAMP, VIRTUAL GENERATED or VIRTUAL STORED

@IANTHEREAL
Copy link
Collaborator

Rest LGTM

@IANTHEREAL
Copy link
Collaborator

LGTM

@amyangfei amyangfei added status/LGT1 One reviewer already commented LGTM and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Feb 21, 2019
csuzhangxc
csuzhangxc approved these changes Feb 21, 2019
View reviewed changes
Copy link
Member

@csuzhangxc csuzhangxc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@csuzhangxc csuzhangxc added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Feb 21, 2019
@amyangfei amyangfei merged commit f4cac6a into pingcap:master Feb 21, 2019
@amyangfei amyangfei deleted the generated-column-ddl branch February 21, 2019 06:20
csuzhangxc pushed a commit to csuzhangxc/dm that referenced this pull request Feb 25, 2019
@amyangfei @csuzhangxc
*: support generated column in loader and binlog replication (pingcap#42
75ed946 
)
lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Apr 6, 2020
@amyangfei
*: support generated column in loader and binlog replication (pingcap#42
6c5eb1e 
)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@IANTHEREAL IANTHEREAL IANTHEREAL left review comments

@csuzhangxc csuzhangxc csuzhangxc approved these changes

Assignees
No one assigned
Labels
priority/normal Minor change, requires approval from ≥1 primary reviewer status/LGT2 Two reviewers already commented LGTM, ready for merge type/bug-fix Bug fix type/qa relate to quality assurance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@amyangfei @IANTHEREAL @csuzhangxc