Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

syncer: add detect server version for upstream and downstream database (#1673) #1693

Merged

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #1673

What problem does this PR solve?

DM didn't detect and print the versions for the upstream and downstream databases.

What is changed and how it works?

Add detect server version for syncer before we start the subtask.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Start a subtask and check the dm-worker's log:
[2021/05/11 12:12:51.380 +08:00] [INFO] [dumpling.go:292] ["found upstream SQL mode"] [task=testdm] [unit=dump] ["SQL mode"=ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION]
[2021/05/11 12:12:51.380 +08:00] [INFO] [dumpling.go:63] ["create dumpling"] [task=testdm] [unit=dump] [config="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":false,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":true,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"127.0.0.1\",\"Port\":3306,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"\",\"LogFile\":\"\",\"LogFormat\":\"\",\"OutputDirPath\":\"./dumped_data.testdm\",\"StatusAddr\":\"\",\"Snapshot\":\"\",\"Consistency\":\"auto\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\"\",\"CsvDelimiter\":\"\",\"Databases\":null,\"Where\":\"\",\"FileType\":\"\",\"ServerInfo\":{\"ServerType\":0,\"ServerVersion\":null},\"Rows\":200000,\"ReadTimeout\":0,\"TiDBMemQuotaQuery\":0,\"FileSize\":67108864,\"StatementSize\":1000000,\"SessionParams\":{},\"Tables\":null}"]
[2021/05/11 12:12:51.696 +08:00] [INFO] [config.go:599] ["detect server type"] [task=testdm] [unit="binlog replication"] [scope=upstream] [type=MySQL]
[2021/05/11 12:12:51.696 +08:00] [INFO] [config.go:618] ["detect server version"] [task=testdm] [unit="binlog replication"] [scope=upstream] [version=5.7.26-log]
[2021/05/11 12:12:51.697 +08:00] [INFO] [config.go:599] ["detect server type"] [task=testdm] [unit="binlog replication"] [scope=downstream] [type=TiDB]
[2021/05/11 12:12:51.697 +08:00] [INFO] [config.go:618] ["detect server version"] [task=testdm] [unit="binlog replication"] [scope=downstream] [version=4.0.11]
[2021/05/11 12:12:52.082 +08:00] [INFO] [checkpoint.go:701] ["create checkpoint schema"] [task=testdm] [unit="binlog replication"] [component="remote checkpoint"] [statement="CREATE SCHEMA IF NOT EXISTS `dm_meta`"]
[2021/05/11 12:12:52.272 +08:00] [INFO] [checkpoint.go:725] ["create checkpoint table"] [task=testdm] [unit="binlog replication"] [component="remote checkpoint"] [statements="[\"CREATE TABLE IF NOT EXISTS `dm_meta`.`testdm_syncer_checkpoint` (\\n\\t\\t\\tid VARCHAR(32) NOT NULL,\\n\\t\\t\\tcp_schema VARCHAR(128) NOT NULL,\\n\\t\\t\\tcp_table VARCHAR(128) NOT NULL,\\n\\t\\t\\tbinlog_name VARCHAR(128),\\n\\t\\t\\tbinlog_pos INT UNSIGNED,\\n\\t\\t\\tbinlog_gtid TEXT,\\n\\t\\t\\texit_safe_binlog_name VARCHAR(128) DEFAULT '',\\n\\t\\t\\texit_safe_binlog_pos INT UNSIGNED DEFAULT 0,\\n\\t\\t\\texit_safe_binlog_gtid TEXT,\\n\\t\\t\\ttable_info JSON NOT NULL,\\n\\t\\t\\tis_global BOOLEAN,\\n\\t\\t\\tcreate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,\\n\\t\\t\\tupdate_time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,\\n\\t\\t\\tUNIQUE KEY uk_id_schema_table (id, cp_schema, cp_table)\\n\\t\\t)\"]"]
[2021/05/11 12:12:52.274 +08:00] [INFO] [subtask.go:223] ["start to run"] [subtask=testdm] [unit=Dump]
[2021/05/11 12:12:52.279 +08:00] [INFO] [config.go:599] ["detect server type"] [task=testdm] [unit=dump] [type=MySQL]
[2021/05/11 12:12:52.279 +08:00] [INFO] [config.go:618] ["detect server version"] [task=testdm] [unit=dump] [version=5.7.26-log]
[2021/05/11 12:12:52.279 +08:00] [INFO] [dump.go:82] ["begin to run Dump"] [task=testdm] [unit=dump] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":false,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":true,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"127.0.0.1\",\"Port\":3306,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"\",\"LogFile\":\"\",\"LogFormat\":\"\",\"OutputDirPath\":\"./dumped_data.testdm\",\"StatusAddr\":\"\",\"Snapshot\":\"\",\"Consistency\":\"flush\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\"\",\"CsvDelimiter\":\"\",\"Databases\":null,\"Where\":\"\",\"FileType\":\"sql\",\"ServerInfo\":{\"ServerType\":1,\"ServerVersion\":\"5.7.26-log\"},\"Rows\":200000,\"ReadTimeout\":0,\"TiDBMemQuotaQuery\":0,\"FileSize\":67108864,\"StatementSize\":1000000,\"SessionParams\":{},\"Tables\":null}"]

Related changes

  • Need to cherry-pick to the release branch

@lichunzhu
Copy link
Contributor

/lgtm

@ti-chi-bot ti-chi-bot added the status/LGT1 One reviewer already commented LGTM label May 14, 2021
Copy link
Collaborator

@GMHDBJD GMHDBJD left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@ti-chi-bot
Copy link
Member Author

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • GMHDBJD
  • lichunzhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels May 14, 2021
@lance6716
Copy link
Collaborator

/merge

@ti-chi-bot
Copy link
Member Author

This pull request has been accepted and is ready to merge.

Commit hash: 8abbc49

@ti-chi-bot
Copy link
Member Author

@ti-chi-bot: Your PR was out of date, I have automatically updated it for you.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot merged commit 20764a0 into pingcap:release-2.0 May 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
size/S status/can-merge status/LGT2 Two reviewers already commented LGTM, ready for merge status/PTAL This PR is ready for review. Add this label back after committing new changes type/cherry-pick-for-release-2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants