client: use ServiceClient for discovery #7611

CabinfeverB · 2023-12-22T08:31:48Z

What problem does this PR solve?

Issue Number: ref #7576

What is changed and how does it work?

remove the leader health flag in the client and maintain it in pdServiceDiscovery.
update leader ServiceClient and forward candidate ServiceClients when updating member info
get gRPC connection from ServiceClients

Check List

Tests

Integration test
Manual test (add detailed scripts or steps below)
replace pd client with this PR in TiDB. Kill pd leader pd-0, qps drops to zero during 2s.

compare to v7.5.0

HA behavior is consistent.

Code changes

Has the configuration change
Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tiup:
Need to cherry-pick to the release branch

Release note

None.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot · 2023-12-22T08:31:51Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

lhy1024
rleungx

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

rleungx · 2023-12-22T08:53:05Z

client/pd_service_discovery.go

+// GetServiceClient returns the leader/primary ServiceClient if it is healthy.
+func (c *pdServiceDiscovery) GetServiceClient() ServiceClient {
+	leaderClient := c.getLeaderServiceClient()
+	if c.option.enableForwarding && !leaderClient.Available() {


What if the leaderClient is nil?

leaderClient will never be nil. If there is no leader in resp, it will return leader address doesn't exist when updateMember.

The name is confusing to me. lol

rleungx · 2023-12-22T08:57:52Z

client/pd_service_discovery.go

-	if addr == oldLeader {
-		return nil
+	oldLeader := c.getLeaderServiceClient()
+	if addr == oldLeader.GetAddress() && oldLeader.GetClientConn() != nil {


Will the oldLeader become nil?

Yes, so GetClientConn() will check nil.

client/pd_service_discovery.go

rleungx · 2023-12-22T09:02:18Z

client/pd_service_discovery.go

+				} else {
+					changed = true
+					conn, err := c.GetOrCreateGRPCConn(addr)
+					follower := newPDServiceClient(addr, leader.GetClientUrls()[0], conn, false)


Yes, it will save the address of leader for forward.

Is it possible len(GetClientUrls)==0?

No, it will return before here

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

codecov · 2023-12-22T09:26:30Z

Codecov Report

Merging #7611 (b2c0b16) into master (b36b725) will increase coverage by 0.00%.
Report is 1 commits behind head on master.
The diff coverage is 91.45%.

❗ Current head b2c0b16 differs from pull request most recent head b2e2fc2. Consider uploading reports for the commit b2e2fc2 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7611   +/-   ##
=======================================
  Coverage   74.77%   74.77%           
=======================================
  Files         459      459           
  Lines       50577    50615   +38     
=======================================
+ Hits        37820    37849   +29     
- Misses       9402     9414   +12     
+ Partials     3355     3352    -3

Flag	Coverage Δ
unittests	`74.77% <91.45%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

CabinfeverB · 2023-12-26T02:02:17Z

add manual test @rleungx

lhy1024 · 2023-12-26T02:33:17Z

client/pd_service_discovery.go

+func (c *pdServiceDiscovery) checkFollowerHealth(ctx context.Context) {
+	c.followers.Range(func(key, value any) bool {
+		// To ensure that the leader's healthy check is not delayed, shorten the duration.
+		ctx, cancel := context.WithTimeout(ctx, MemberHealthCheckInterval/3)


why not c.option.timeout?

MemberHealthCheckInterval is shorter than defaultPDTimeout

CabinfeverB · 2023-12-26T06:20:59Z

/merge

ti-chi-bot · 2023-12-26T06:21:01Z

@CabinfeverB: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-12-26T06:21:04Z

This pull request has been accepted and is ready to merge.

Commit hash: b2c0b16

ti-chi-bot · 2023-12-26T06:21:15Z

@CabinfeverB: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

use ServiceClient in pd discovery

82657d4

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot bot added the release-note-none Denotes a PR that doesn't merit a release note. label Dec 22, 2023

ti-chi-bot bot requested review from disksing and JmPotato December 22, 2023 08:31

ti-chi-bot bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 22, 2023

CabinfeverB requested review from lhy1024 and rleungx and removed request for disksing December 22, 2023 08:32

CabinfeverB mentioned this pull request Dec 22, 2023

*: follower support to handle GetRegion and other region api #7432

Merged

rleungx reviewed Dec 22, 2023

View reviewed changes

use ServiceClient in pd discovery

a2d7dfb

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

CabinfeverB mentioned this pull request Dec 22, 2023

Enhance ServiceDiscovery to improve client high availability and load balance #7576

Open

4 tasks

address comment

b2c0b16

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

rleungx approved these changes Dec 25, 2023

View reviewed changes

ti-chi-bot bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 25, 2023

lhy1024 approved these changes Dec 26, 2023

View reviewed changes

ti-chi-bot bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 26, 2023

ti-chi-bot bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 26, 2023

Merge branch 'master' into follower/refactor_discovery_2

b2e2fc2

ti-chi-bot bot merged commit 4d985b2 into tikv:master Dec 26, 2023
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

client: use ServiceClient for discovery #7611

client: use ServiceClient for discovery #7611

CabinfeverB commented Dec 22, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 22, 2023 •

edited

Loading

rleungx Dec 22, 2023

CabinfeverB Dec 22, 2023

rleungx Dec 22, 2023

rleungx Dec 22, 2023

CabinfeverB Dec 22, 2023

rleungx Dec 22, 2023

CabinfeverB Dec 22, 2023

rleungx Dec 22, 2023

CabinfeverB Dec 23, 2023 •

edited

Loading

codecov bot commented Dec 22, 2023 •

edited

Loading

CabinfeverB commented Dec 26, 2023

lhy1024 Dec 26, 2023

CabinfeverB Dec 26, 2023

CabinfeverB commented Dec 26, 2023

ti-chi-bot bot commented Dec 26, 2023

ti-chi-bot bot commented Dec 26, 2023

ti-chi-bot bot commented Dec 26, 2023

client: use ServiceClient for discovery #7611

client: use ServiceClient for discovery #7611

Conversation

CabinfeverB commented Dec 22, 2023 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot bot commented Dec 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CabinfeverB Dec 23, 2023 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Dec 22, 2023 • edited Loading

Codecov Report

CabinfeverB commented Dec 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CabinfeverB commented Dec 26, 2023

ti-chi-bot bot commented Dec 26, 2023

ti-chi-bot bot commented Dec 26, 2023

ti-chi-bot bot commented Dec 26, 2023

CabinfeverB commented Dec 22, 2023 •

edited

Loading

ti-chi-bot bot commented Dec 22, 2023 •

edited

Loading

CabinfeverB Dec 23, 2023 •

edited

Loading

codecov bot commented Dec 22, 2023 •

edited

Loading