*(all): update dependencies on tidb and pd. #8126

asddongmen · 2023-01-29T03:19:29Z

What problem does this PR solve?

Issue Number: close #8110 #8115

What is changed and how it works?

Upgrade tiflow's package dependencies on tidb, pd, and tikv to properly call the pdClient interface.

TiCDC behavior after PR merge:

TiCDC version > 6.5.0:
Compatible with both low and high version PDs.
Can get the correct sourceID.
TiCDC version = 6.5.0:
Cannot get the correct sourceID from higher version PDs (v>6.5.0).
No error, but no effect on data correctness.
TiCDC version < 6.5.0:
The interface to get the sourceID is not called.
No compatibility issues.

Check List

Tests

Manual test (add detailed scripts or steps below)
Unit test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix the problem causing changefeed creation to fail due to inability to retrieve the sourceID from PD.

ti-chi-bot · 2023-01-29T03:19:30Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

hi-rustin
zhaoxinyu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

pkg/pdutil/utils.go

asddongmen · 2023-01-29T05:29:25Z

/run-integration-tests

HuSharp · 2023-01-29T06:59:57Z

I see that for ci The job was not started because recent account payments have failed or your spending limit needs to be increased.....

asddongmen · 2023-01-30T03:33:49Z

/run-verify

asddongmen · 2023-01-30T07:48:30Z

/hold

asddongmen · 2023-02-06T15:22:11Z

/run-kafka-integration-test

lance6716 · 2023-02-06T20:24:55Z

/run-dm-integration-test

Rustin170506

Thanks for working on this.

Makefile

cdc/api/v2/changefeed_test.go

cdc/entry/schema_storage_test.go

pkg/pdutil/utils.go

pkg/pdutil/utils_test.go

Makefile

cdc/entry/schema/snapshot.go

overvenus · 2023-02-07T06:04:08Z

tests/integration_tests/multi_source/main.go

@@ -194,8 +194,8 @@ func truncateDDL(ctx context.Context, db *sql.DB) {

 func ignoreableError(err error) bool {
 knownErrorList := []string{
- "Error 1146:", // table doesn't exist
- "Error 1049", // database doesn't exist
+ "Error 1146", // table doesn't exist


Why change the test?

After we upgrade go-sql-driver, the error message is changed, for example

Error 1142 (42000): ALTER command denied

cdc/entry/schema/snapshot.go

overvenus · 2023-02-07T08:26:37Z

Makefile

 ARCH := "$(shell uname -s)"
 LINUX := "Linux"
 MAC := "Darwin"
+ifeq (${ARCH}, ${LINUX})
+ CGO := 0


What if someone cross compile macOS TiCDC on a linux machine?

I think only GOOS matters.

Thank you! You are right, only GOOS matters, but no matter set CGO_ENABLED=0 or CGO_ENABLED=1 runGOOS=darwin make cdc fails.
I revert related changes and run the following commands:

➜ ticdc git:(fix-get-source-id) uname Linux ➜ ticdc git:(fix-get-source-id) uname -m x86_64 ➜ ticdc git:(fix-get-source-id) CGO_ENABLED=0 GOOS=darwin make cdc GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v6.5.0-master-dirty" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2023-02-07 09:41:40" -X "github.com/pingcap/tiflow/pkg/version.GitHash=03e6f4650e98fe8b47788622844fc1a5f9272ead" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=fix-get-source-id" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.19.3 linux/amd64"' -o bin/cdc ./cmd/cdc/main.go # github.com/cloudfoundry/gosigar vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:20:12: cpuUsage.Get undefined (type Cpu has no field or method Get) vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:30:14: cpuUsage.Get undefined (type Cpu has no field or method Get) vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:49:11: l.Get undefined (type LoadAverage has no field or method Get) vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:55:11: m.Get undefined (type Mem has no field or method Get) vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:61:11: m.GetIgnoringCGroups undefined (type Mem has no field or method GetIgnoringCGroups) vendor/github.com/cloudfoundry/gosigar/concrete_sigar.go:67:11: s.Get undefined (type Swap has no field or method Get) vendor/github.com/cloudfoundry/gosigar/sigar_shared.go:12:21: procTime.Get undefined (type *ProcTime has no field or method Get) ^Cmake: *** [cdc] Interrupt ➜ ticdc git:(fix-get-source-id) ✗ CGO_ENABLED=1 GOOS=darwin make cdc GO111MODULE=on go build -trimpath -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v6.5.0-master-dirty" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2023-02-07 09:41:32" -X "github.com/pingcap/tiflow/pkg/version.GitHash=03e6f4650e98fe8b47788622844fc1a5f9272ead" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=fix-get-source-id" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.19.3 linux/amd64"' -o bin/cdc ./cmd/cdc/main.go # runtime/cgo gcc: error: x86_64: No such file or directory gcc: error: unrecognized command line option '-arch' make: *** [cdc] Error 2

Do you have any idea to fix this problem?

pingcap/tidb#41165

use CGO_ENABLED=1 is the only way to build on darwin.

pingcap/tidb#41165

use CGO_ENABLED=1 is the only way to build on darwin.

Yes, but can't build a darwin cdc on linux machine.

Makefile

Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com>

asddongmen · 2023-02-07T14:28:36Z

/merge

ti-chi-bot · 2023-02-07T14:28:39Z

This pull request has been accepted and is ready to merge.

Commit hash: 3fd5f58

ti-chi-bot · 2023-02-07T14:28:53Z

@asddongmen: Your PR was out of date, I have automatically updated it for you.

At the same time I will also trigger all tests for you:

/run-all-tests

trigger some heavy tests which will not run always when PR updated.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

asddongmen · 2023-02-08T02:03:25Z

/run-dm-integration-test

close pingcap#8110, ref pingcap#8115 (cherry picked from commit 5a38ed9)

fix get source id error

dde7691

asddongmen changed the title ~~[WIP] ticdc: fix get source id error~~ ticdc: fix get source id error Jan 29, 2023

asddongmen requested review from overvenus and lance6716 January 29, 2023 03:37

overvenus reviewed Jan 29, 2023

View reviewed changes

pkg/pdutil/utils.go Outdated Show resolved Hide resolved

asddongmen added 3 commits January 29, 2023 13:23

add unit test

f8011ce

add unit test

022862a

fix fmt

72c0087

asddongmen added the status/ptal Could you please take a look? label Jan 29, 2023

asddongmen added 4 commits January 29, 2023 15:06

fix ut

053c101

fix ut

34b01cd

fix ut

9c47ff4

fix ut

dee9636

asddongmen added 2 commits January 30, 2023 14:05

add arch check to makefile

cb0aa87

format makefile

9a0d3db

ti-chi-bot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jan 30, 2023

asddongmen added 2 commits February 3, 2023 15:43

merge upstream/master

7bc84de

update dependencies

37f27f7

asddongmen changed the title ~~ticdc: fix get source id error~~ ticdc: update dependencies on tidb and pd. Feb 7, 2023

Rustin170506 reviewed Feb 7, 2023

View reviewed changes

Rustin170506 changed the title ~~ticdc: update dependencies on tidb and pd.~~ *(all): update dependencies on tidb and pd. Feb 7, 2023

zhaoxinyu reviewed Feb 7, 2023

View reviewed changes

Makefile Show resolved Hide resolved

zhaoxinyu reviewed Feb 7, 2023

View reviewed changes

cdc/entry/schema/snapshot.go Outdated Show resolved Hide resolved

overvenus reviewed Feb 7, 2023

View reviewed changes

zhaoxinyu reviewed Feb 7, 2023

View reviewed changes

cdc/entry/schema/snapshot.go Outdated Show resolved Hide resolved

asddongmen added 3 commits February 7, 2023 15:26

address comments

c42dab2

address comments again

4b2e19b

remove debug log

03e6f46

zhaoxinyu approved these changes Feb 7, 2023

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Feb 7, 2023

overvenus reviewed Feb 7, 2023

View reviewed changes

address comments

f1628fd

Rustin170506 approved these changes Feb 7, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Feb 7, 2023

Update Makefile

3fd5f58

Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Feb 7, 2023

Merge branch 'master' into fix-get-source-id

b35f4e9

ti-chi-bot merged commit 5a38ed9 into pingcap:master Feb 8, 2023

asddongmen mentioned this pull request Feb 8, 2023

cdc failed to run on nightly version #8115

Closed

asddongmen added a commit to asddongmen/tiflow that referenced this pull request Feb 15, 2023

*(all): update dependencies on tidb and pd. (pingcap#8126)

e675bd4

close pingcap#8110, ref pingcap#8115 (cherry picked from commit 5a38ed9)

asddongmen deleted the fix-get-source-id branch May 18, 2023 05:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*(all): update dependencies on tidb and pd. #8126

*(all): update dependencies on tidb and pd. #8126

asddongmen commented Jan 29, 2023 •

edited

Loading

ti-chi-bot commented Jan 29, 2023 •

edited

Loading

asddongmen commented Jan 29, 2023

HuSharp commented Jan 29, 2023 •

edited

Loading

asddongmen commented Jan 30, 2023

asddongmen commented Jan 30, 2023

asddongmen commented Feb 6, 2023

lance6716 commented Feb 6, 2023

Rustin170506 left a comment

overvenus Feb 7, 2023

lance6716 Feb 7, 2023

overvenus Feb 7, 2023

overvenus Feb 7, 2023

asddongmen Feb 7, 2023 •

edited

Loading

lance6716 Feb 7, 2023

asddongmen Feb 8, 2023

asddongmen commented Feb 7, 2023

ti-chi-bot commented Feb 7, 2023

ti-chi-bot commented Feb 7, 2023

asddongmen commented Feb 8, 2023

*(all): update dependencies on tidb and pd. #8126

*(all): update dependencies on tidb and pd. #8126

Conversation

asddongmen commented Jan 29, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Tests

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

ti-chi-bot commented Jan 29, 2023 • edited Loading

asddongmen commented Jan 29, 2023

HuSharp commented Jan 29, 2023 • edited Loading

asddongmen commented Jan 30, 2023

asddongmen commented Jan 30, 2023

asddongmen commented Feb 6, 2023

lance6716 commented Feb 6, 2023

Rustin170506 left a comment

Choose a reason for hiding this comment

overvenus Feb 7, 2023

Choose a reason for hiding this comment

lance6716 Feb 7, 2023

Choose a reason for hiding this comment

overvenus Feb 7, 2023

Choose a reason for hiding this comment

overvenus Feb 7, 2023

Choose a reason for hiding this comment

asddongmen Feb 7, 2023 • edited Loading

Choose a reason for hiding this comment

lance6716 Feb 7, 2023

Choose a reason for hiding this comment

asddongmen Feb 8, 2023

Choose a reason for hiding this comment

asddongmen commented Feb 7, 2023

ti-chi-bot commented Feb 7, 2023

ti-chi-bot commented Feb 7, 2023

asddongmen commented Feb 8, 2023

asddongmen commented Jan 29, 2023 •

edited

Loading

ti-chi-bot commented Jan 29, 2023 •

edited

Loading

HuSharp commented Jan 29, 2023 •

edited

Loading

asddongmen Feb 7, 2023 •

edited

Loading