Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd, pd (ticdc): refine pdClient and etcdClient initialization #9661

Merged

Conversation

asddongmen
Copy link
Contributor

@asddongmen asddongmen commented Aug 31, 2023

What problem does this PR solve?

Issue Number: close #9565 #9532
ref #9054 ref #8808

What is changed and how it works?

This PR can decrease the probability of cdc capture restarting during pd network isolation and increase the success rate of restarts, resulting in reduced delay for changefeed in this scenario. I tested it on tisapce and found that with this PR, network isolation no longer causes cdc capture and server to exit. (In theory, it is still possible for cdc to exit, but I did not observe it in my testing. Moreover, if cdc exits, the success rate of restarts will be greatly improved.)

The specific approach is:

  1. Reduce the number of pdClients that need to be created when cdc server starts from 4 to 2. This can decrease the probability of cdc server failing to start due to pdClient creation failure.
  2. After successfully creating the first pdClient, use it to obtain all online pd addresses and update the pd addresses passed to server startup parameters. After this, new addresses will be used for creating other pdClients and etcdClients.
  3. When creating etcdClient, a goroutine will be started to maintain its internal address update logic, which is basically copied from pd. Reference: https://github.com/tikv/pd/blob/master/pkg/utils/etcdutil/etcdutil.go#L249

Check List

Tests

  • Manual test (add detailed scripts or steps below)
  1. Deploy a TiDB cluster with 3 PD nodes and 2 TiCDC nodes, create a synchronization task, and continuously write to the upstream TiDB.
  2. Use the following iptables command to inject network isolation into any PD node.
sudo iptables -A OUTPUT -p tcp --dport 2379 -j DROP
  1. During the test, observe the server status and synchronization delay of CDC, both of which are normal.

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug that may cause cdc restart many time when there is a PD network isolation in the upstream PD cluster.

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. do-not-merge/needs-triage-completed needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. and removed do-not-merge/needs-linked-issue do-not-merge/needs-triage-completed labels Aug 31, 2023
@asddongmen
Copy link
Contributor Author

/retest

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Sep 5, 2023
@asddongmen
Copy link
Contributor Author

/retest

@@ -81,6 +82,7 @@ type captureImpl struct {
liveness model.Liveness
config *config.ServerConfig

pdClient pd.Client
pdEndpoints []string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pdClient and pdEndpoint are maintained at the same time, this looks weird

pkg/etcd/client.go Outdated Show resolved Hide resolved
@asddongmen
Copy link
Contributor Author

/retest

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 5, 2023
@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 5, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,sdojjy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 5, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-09-05 01:19:30.529838233 +0000 UTC m=+2408335.078854217: ☑️ agreed by sdojjy.
  • 2023-09-05 03:04:52.312635029 +0000 UTC m=+2414656.861651016: ☑️ agreed by 3AceShowHand.

@ti-chi-bot ti-chi-bot bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 7, 2023
@asddongmen
Copy link
Contributor Author

/retest

@asddongmen
Copy link
Contributor Author

/test all

@ti-chi-bot ti-chi-bot bot merged commit 4452688 into pingcap:master Sep 8, 2023
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #9708.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Sep 8, 2023
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #9709.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

two of ticdc restart repeatedly after injection network partition between pdleader and pd followers
4 participants