dm/test: swap upstream and downstream of sync_diff in some cases #3413

lance6716 · 2021-11-11T09:15:33Z

What problem does this PR solve?

close https://github.com/pingcap/ticdc/issues/3335
close https://github.com/pingcap/ticdc/issues/3409
close https://github.com/pingcap/ticdc/issues/3394

also close https://github.com/pingcap/ticdc/pull/3405

What is changed and how it works?

as title

Check List

Tests

No code

Code changes

Side effects

Related changes

Need to cherry-pick to the release branch

Release note

`None`.

ti-chi-bot · 2021-11-11T09:15:34Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Ehco1996
lichunzhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

lance6716 · 2021-11-11T09:35:06Z

/run-all-tests

GMHDBJD · 2021-11-11T09:55:46Z

dm_syncer
["failed to initialize diff process"] [error="no table need to be compared"] [errorVerbose="no table need to be compared\ngithub.com/pingcap/tidb-tools/sync_diff_inspector/source.NewSources\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/source/source.go:150\nmain.(*Diff).init\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/diff.go:139\nmain.NewDiff\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/diff.go:97\nmain.checkSyncState\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:92\nmain.main\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:79\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [stack="main.checkSyncState\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:95\nmain.main\n\t/home/jenkins/agent/workspace/optimization-build-tidb-linux-amd/go/src/github.com/pingcap/tidb-tools/sync_diff_inspector/main.go:79\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225"]

lance6716 · 2021-11-11T11:05:17Z

/run-all-tests

lance6716 · 2021-11-11T12:29:14Z

/run-dm-integration-test

dm/tests/full_mode/run.sh

Ehco1996 · 2021-11-12T00:06:18Z

/run-dm-integration-tests

Ehco1996

restLGTM

dm/tests/adjust_gtid/run.sh

Ehco1996 · 2021-11-12T00:26:43Z

dm/tests/dm_syncer/run.sh

@@ -71,7 +71,7 @@ function run() {
 	run_dm_syncer $WORK_DIR/syncer2 $WORK_DIR/dm-syncer-2.toml $meta_file --syncer-config-format syncer2

 	# wait for dm_syncer to init and start
-	sleep 5
+	sleep 10


in our doc: Syncer is no longer maintained. Its features are completely superseded by TiDB Data Migration. It is strongly recommended that you use TiDB Data Migration instead

just a question: do we have plan to delete the code releated to dm-sycner from our codebase and syncer-bin from tidb-enterprise-tools?

also cc @sunzhaoyang

dm/tests/full_mode/run.sh

lance6716 · 2021-11-12T02:14:56Z

/run-all-tests

lance6716 · 2021-11-12T02:36:24Z

/merge

ti-chi-bot · 2021-11-12T02:36:27Z

This pull request has been accepted and is ready to merge.

Commit hash: 5067ec5

Ehco1996 · 2021-11-12T02:51:05Z

[2021-11-12T02:43:45.884Z] Exception:

[2021-11-12T02:43:45.884Z] Traceback (most recent call last):

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main

[2021-11-12T02:43:45.884Z]     status = self.run(options, args)

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 307, in run

[2021-11-12T02:43:45.884Z]     requirement_set.prepare_files(finder)

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 370, in prepare_files

[2021-11-12T02:43:45.884Z]     ignore_dependencies=self.ignore_dependencies))

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 587, in _prepare_file

[2021-11-12T02:43:45.884Z]     session=self.session, hashes=hashes)

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/download.py", line 811, in unpack_url

[2021-11-12T02:43:45.884Z]     hashes=hashes

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/download.py", line 650, in unpack_http_url

[2021-11-12T02:43:45.884Z]     hashes)

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/download.py", line 892, in _download_http_url

[2021-11-12T02:43:45.884Z]     _download_url(resp, link, content_file, hashes)

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/download.py", line 596, in _download_url

[2021-11-12T02:43:45.884Z]     hashes.check_against_chunks(downloaded_chunks)

[2021-11-12T02:43:45.884Z]   File "/usr/lib/python2.7/site-packages/pip/utils/hashes.py", line 49, in check_against_chunks

[2021-11-12T02:43:45.884Z]     for chunk in chunks:

[2021-11-12T02:43:45.885Z]   File "/usr/lib/python2.7/site-packages/pip/download.py", line 564, in written_chunks

[2021-11-12T02:43:45.885Z]     for chunk in chunks:

[2021-11-12T02:43:45.885Z]   File "/usr/lib/python2.7/site-packages/pip/download.py", line 553, in resp_read

[2021-11-12T02:43:45.885Z]     decode_content=False):

[2021-11-12T02:43:45.885Z]   File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 353, in stream

[2021-11-12T02:43:45.885Z]     data = self.read(amt=amt, decode_content=decode_content)

[2021-11-12T02:43:45.885Z]   File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 320, in read

[2021-11-12T02:43:45.885Z]     flush_decoder = True

[2021-11-12T02:43:45.885Z]   File "/usr/lib64/python2.7/contextlib.py", line 35, in __exit__

[2021-11-12T02:43:45.885Z]     self.gen.throw(type, value, traceback)

[2021-11-12T02:43:45.885Z]   File "/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/response.py", line 242, in _error_catcher

[2021-11-12T02:43:45.885Z]     raise ReadTimeoutError(self._pool, None, 'Read timed out.')

[2021-11-12T02:43:45.885Z] ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

i will add retry for this

lance6716 · 2021-11-12T03:09:08Z

/run-all-tests

lance6716 · 2021-11-12T03:34:20Z

/merge

lance6716 · 2021-11-12T03:53:17Z

/run-integration-test

lance6716 · 2021-11-12T04:23:19Z

/run-integration-test

upload coverage error...

lance6716 · 2021-11-12T05:51:18Z

/run-integration-test

…gcap#3413)

…ses (#3413)" This reverts commit 35102b5.

…ses (pingcap#3413)" This reverts commit 35102b5.

dm/test: swap upstream and downstream of sync_diff in some cases

d043431

lance6716 added 2 commits November 11, 2021 17:17

remove a config

30450fa

Merge branch 'master' of github.com:pingcap/ticdc into fix-sync_diff

6cdb3ed

lance6716 requested review from Ehco1996, lichunzhu and GMHDBJD November 11, 2021 09:18

lance6716 mentioned this pull request Nov 11, 2021

dm: add retry in it full_mode test #3405

Closed

increase slepp time

668759d

lichunzhu approved these changes Nov 11, 2021

View reviewed changes

dm/tests/full_mode/run.sh Show resolved Hide resolved

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 11, 2021

lance6716 commented Nov 11, 2021

View reviewed changes

dm/tests/full_mode/run.sh Outdated Show resolved Hide resolved

Update dm/tests/full_mode/run.sh

0d1888f

Ehco1996 added the area/dm Issues or PRs related to DM. label Nov 12, 2021

Merge branch 'master' into fix-sync_diff

5067ec5

Ehco1996 reviewed Nov 12, 2021

View reviewed changes

Ehco1996 approved these changes Nov 12, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 12, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 12, 2021

Merge branch 'master' into fix-sync_diff

5d55c32

ti-chi-bot merged commit 35102b5 into pingcap:master Nov 12, 2021

lance6716 mentioned this pull request Nov 15, 2021

sync-diff: add table match check pingcap/tidb-tools#548

Merged

glorv pushed a commit to glorv/ticdc that referenced this pull request Nov 17, 2021

dm/test: swap upstream and downstream of sync_diff in some cases (pin…

d55eb26

…gcap#3413)

glorv mentioned this pull request Nov 17, 2021

dm: revert dm#2047 #3491

Merged

lance6716 mentioned this pull request Dec 31, 2021

sync-diff will not report error when upstream has table but downstream doesn't pingcap/tidb-tools#583

Closed

lance6716 added a commit that referenced this pull request Jan 18, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca…

16b99c7

…ses (#3413)" This reverts commit 35102b5.

lance6716 added a commit to lance6716/ticdc that referenced this pull request Jan 18, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca…

c540a1a

…ses (pingcap#3413)" This reverts commit 35102b5.

lance6716 mentioned this pull request Jan 18, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca… #4382

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 19, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca…

e170a22

…ses (pingcap#3413)" This reverts commit 35102b5.

ti-chi-bot mentioned this pull request Jan 19, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca… (#4382) #4398

Merged

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 19, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca…

1971caa

…ses (pingcap#3413)" This reverts commit 35102b5.

ti-chi-bot mentioned this pull request Jan 19, 2022

Revert "dm/test: swap upstream and downstream of sync_diff in some ca… (#4382) #4399

Merged

lance6716 deleted the fix-sync_diff branch October 13, 2022 08:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dm/test: swap upstream and downstream of sync_diff in some cases #3413

dm/test: swap upstream and downstream of sync_diff in some cases #3413

lance6716 commented Nov 11, 2021 •

edited by Ehco1996

Loading

ti-chi-bot commented Nov 11, 2021 •

edited

Loading

lance6716 commented Nov 11, 2021

GMHDBJD commented Nov 11, 2021

lance6716 commented Nov 11, 2021

lance6716 commented Nov 11, 2021

Ehco1996 commented Nov 12, 2021

Ehco1996 left a comment

Ehco1996 Nov 12, 2021 •

edited

Loading

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

ti-chi-bot commented Nov 12, 2021

Ehco1996 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

dm/test: swap upstream and downstream of sync_diff in some cases #3413

dm/test: swap upstream and downstream of sync_diff in some cases #3413

Conversation

lance6716 commented Nov 11, 2021 • edited by Ehco1996 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Nov 11, 2021 • edited Loading

lance6716 commented Nov 11, 2021

GMHDBJD commented Nov 11, 2021

lance6716 commented Nov 11, 2021

lance6716 commented Nov 11, 2021

Ehco1996 commented Nov 12, 2021

Ehco1996 left a comment

Choose a reason for hiding this comment

Ehco1996 Nov 12, 2021 • edited Loading

Choose a reason for hiding this comment

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

ti-chi-bot commented Nov 12, 2021

Ehco1996 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 12, 2021

lance6716 commented Nov 11, 2021 •

edited by Ehco1996

Loading

ti-chi-bot commented Nov 11, 2021 •

edited

Loading

Ehco1996 Nov 12, 2021 •

edited

Loading