Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd, pd (ticdc): refine pdClient and etcdClient initialization (#9661) #9709

Merged

Conversation

ti-chi-bot
Copy link
Member

This is an automated cherry-pick of #9661

What problem does this PR solve?

Issue Number: close #9565 #9532
ref #9054

What is changed and how it works?

This PR can decrease the probability of cdc capture restarting during pd network isolation and increase the success rate of restarts, resulting in reduced delay for changefeed in this scenario. I tested it on tisapce and found that with this PR, network isolation no longer causes cdc capture and server to exit. (In theory, it is still possible for cdc to exit, but I did not observe it in my testing. Moreover, if cdc exits, the success rate of restarts will be greatly improved.)

The specific approach is:

  1. Reduce the number of pdClients that need to be created when cdc server starts from 4 to 2. This can decrease the probability of cdc server failing to start due to pdClient creation failure.
  2. After successfully creating the first pdClient, use it to obtain all online pd addresses and update the pd addresses passed to server startup parameters. After this, new addresses will be used for creating other pdClients and etcdClients.
  3. When creating etcdClient, a goroutine will be started to maintain its internal address update logic, which is basically copied from pd. Reference: https://github.com/tikv/pd/blob/master/pkg/utils/etcdutil/etcdutil.go#L249

Check List

Tests

  • Manual test (add detailed scripts or steps below)
  1. Deploy a TiDB cluster with 3 PD nodes and 2 TiCDC nodes, create a synchronization task, and continuously write to the upstream TiDB.
  2. Use the following iptables command to inject network isolation into any PD node.
sudo iptables -A OUTPUT -p tcp --dport 2379 -j DROP
  1. During the test, observe the server status and synchronization delay of CDC, both of which are normal.

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a bug that may cause cdc restart many time when there is a PD network isolation in the upstream PD cluster.

@ti-chi-bot ti-chi-bot added lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR. labels Sep 8, 2023
@ti-chi-bot ti-chi-bot added the cherry-pick-approved Cherry pick PR approved by release team. label Sep 11, 2023
@asddongmen
Copy link
Contributor

/retest

@ti-chi-bot ti-chi-bot bot added the approved label Sep 19, 2023
@asddongmen
Copy link
Contributor

/lgtm

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Sep 19, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot merged commit f8ca1ca into pingcap:release-7.1 Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved cherry-pick-approved Cherry pick PR approved by release team. lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. type/cherry-pick-for-release-7.1 This PR is cherry-picked to release-7.1 from a source PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants