resource_manager: implement token assignment in server #5809

CabinfeverB · 2022-12-29T12:14:29Z

Signed-off-by: Cabinfever_B cabinfeveroier@gmail.com

What problem does this PR solve?

Issue Number: Close #5822

What is changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has configuration change
Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tiup:
Need to cherry-pick to the release branch

Release note

Please add a release note.

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with None.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot · 2022-12-29T12:14:31Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

JmPotato
nolouch

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

lhy1024

roken? token?

codecov · 2022-12-29T12:23:36Z

Codecov Report

Base: 75.40% // Head: 75.45% // Increases project coverage by +0.04% 🎉

Coverage data is based on head (3865ab3) compared to base (1b778f2).
Patch coverage: 49.12% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5809      +/-   ##
==========================================
+ Coverage   75.40%   75.45%   +0.04%     
==========================================
  Files         338      338              
  Lines       33774    33859      +85     
==========================================
+ Hits        25468    25548      +80     
- Misses       6113     6120       +7     
+ Partials     2193     2191       -2

Flag	Coverage Δ
unittests	`75.45% <49.12%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/mcs/resource_manager/server/grpc_service.go	`40.57% <0.00%> (-35.10%)`	⬇️
pkg/mcs/resource_manager/server/resource_group.go	`59.81% <0.00%> (-1.14%)`	⬇️
pkg/mcs/resource_manager/server/token_bukets.go	`89.18% <87.50%> (+50.05%)`	⬆️
pkg/utils/metricutil/metricutil.go	`82.75% <0.00%> (-10.35%)`	⬇️
pkg/utils/tempurl/tempurl.go	`60.00% <0.00%> (-10.00%)`	⬇️
...erver/config/service_middleware_persist_options.go	`91.66% <0.00%> (-8.34%)`	⬇️
server/storage/storage.go	`67.30% <0.00%> (-3.85%)`	⬇️
server/schedulers/random_merge.go	`62.50% <0.00%> (-3.13%)`	⬇️
pkg/election/leadership.go	`77.31% <0.00%> (-3.10%)`	⬇️
server/server.go	`74.61% <0.00%> (-1.03%)`	⬇️
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

nolouch · 2023-01-03T06:21:10Z

pkg/mcs/resource_manager/server/token_bukets.go

 type GroupTokenBucket struct {
 	*rmpb.TokenBucket `json:"token_bucket,omitempty"`
 	Consumption       *rmpb.TokenBucketsRequest `json:"consumption,omitempty"`
 	LastUpdate        *time.Time                `json:"last_update,omitempty"`
 	Initialized       bool                      `json:"initialized"`
+	LoanExpireTime    *time.Time                `json:"loan_time,omitempty"`


Add comments about the loan.

nolouch

plz link the issue.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com> impl token request and fix update bug Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

glorv · 2023-01-04T08:30:33Z

pkg/mcs/resource_manager/server/grpc_service.go

+		for _, req := range request.Requests {
+			rg := s.manager.GetResourceGroup(req.ResourceGroupName)
+			if rg == nil {
+				return errors.New("resource group not found")


A logical error should not exit the main loop. Should set a error message in the response instead.

glorv · 2023-01-04T08:31:07Z

pkg/mcs/resource_manager/server/grpc_service.go

+					}
+				}
+			case rmpb.GroupMode_NativeMode:
+				return errors.New("not supports the resource type")


nolouch · 2023-01-05T08:20:10Z

pkg/mcs/resource_manager/server/token_bukets.go

+	var res rmpb.TokenBucket
+	res.Settings = &rmpb.TokenLimitSettings{}
+	// TODO: consider the shares for dispatch the fill rate
+	res.Settings.Fillrate = 0


I think it's used for the token server unavailable in abnormal situation.

nolouch · 2023-01-05T08:30:46Z

pkg/mcs/resource_manager/server/token_bukets.go

+	} else {
+		et := t.LastUpdate.Add(t.LoanMaxPeriod)
+		t.LoanExpireTime = &et
+		periodFilled = float64(t.Settings.Fillrate) * (1 - loanReserveRatio) * t.LoanMaxPeriod.Seconds()


Do we need to ensure LoadMaxPeriod period greater than targetPeriodMs ?

and why not periodFilled = float64(t.Settings.Fillrate) * (1 - loanReserveRatio) * trickleTime ？

periodFilled is the number of tokens that can be allocated in whole loan period, so I think we should use LoanMaxPeriod

I think it will make the tokens not be graceful consumed with Fillrate in the loan period. Client will consume more tokens in target period(loan expire period> target period)

I guess that you want the consumption curve to be smoother. To fit this situation, we can use shorter LoanMaxPeriod. We can use long LoanMaxPeriod for high credit or important resource group, so they can respond more promptly to qps peaks.

nolouch · 2023-01-05T08:34:04Z

pkg/mcs/resource_manager/server/token_bukets.go

+		periodFilled = float64(t.Settings.Fillrate) * (1 - loanReserveRatio) * t.LoanMaxPeriod.Seconds()
+	}
+	periodFilled += t.Tokens
+	if periodFilled <= float64(t.Settings.Fillrate)*loanReserveRatio {


need * trickleTime?

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

nolouch · 2023-01-06T08:03:48Z

@CabinfeverB Has some conflicts.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

nolouch · 2023-01-09T08:50:49Z

pkg/mcs/resource_manager/server/token_bukets.go

-// Request requests tokens from the token bucket.
-func (t *GroupTokenBucket) Request(
+// request requests tokens from the token bucket.
+func (t *GroupTokenBucket) request(


I think request and update can be one function, also reduce the code functions.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

JmPotato · 2023-01-10T07:28:54Z

pkg/mcs/resource_manager/server/grpc_service.go

+					switch re.Type {
+					case rmpb.RequestUnitType_RRU:
+						tokens := rg.RequestRRU(now, re.Value, targetPeriodMs)
+						resp.GrantedRUTokens = append(resp.GrantedRUTokens, tokens)
+					case rmpb.RequestUnitType_WRU:
+						tokens := rg.RequestWRU(now, re.Value, targetPeriodMs)
+						resp.GrantedRUTokens = append(resp.GrantedRUTokens, tokens)
+					}


Suggested change

switch re.Type {

case rmpb.RequestUnitType_RRU:

tokens := rg.RequestRRU(now, re.Value, targetPeriodMs)

resp.GrantedRUTokens = append(resp.GrantedRUTokens, tokens)

case rmpb.RequestUnitType_WRU:

tokens := rg.RequestWRU(now, re.Value, targetPeriodMs)

resp.GrantedRUTokens = append(resp.GrantedRUTokens, tokens)

}

switch re.Type {

case rmpb.RequestUnitType_RRU:

tokens := rg.RequestRRU(now, re.Value, targetPeriodMs)

case rmpb.RequestUnitType_WRU:

tokens := rg.RequestWRU(now, re.Value, targetPeriodMs)

}

resp.GrantedRUTokens = append(resp.GrantedRUTokens, tokens)

JmPotato · 2023-01-10T07:31:16Z

pkg/mcs/resource_manager/server/token_bukets.go


 // GroupTokenBucket is a token bucket for a resource group.
+// TODO: statistics Consumption


Suggested change

// TODO: statistics Consumption

// TODO: statistics consumption @JmPotato

nolouch

rest LGTM

nolouch · 2023-01-10T09:17:25Z

pkg/mcs/resource_manager/server/token_bukets.go

+	// MaxTokens limits the number of tokens that can be accumulated
+	MaxTokens float64 `json:"max_tokens,omitempty"`
+
+	Consumption *rmpb.TokenBucketsRequest `json:"consumption,omitempty"`


Suggested change

Consumption *rmpb.TokenBucketsRequest `json:"consumption,omitempty"`

Consumption *rmpb.Consumption `json:"consumption,omitempty"`

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

JmPotato · 2023-01-10T10:10:37Z

/merge

ti-chi-bot · 2023-01-10T10:10:39Z

@JmPotato: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-01-10T10:10:41Z

This pull request has been accepted and is ready to merge.

Commit hash: 3896b95

ti-chi-bot · 2023-01-10T10:10:54Z

@CabinfeverB: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

impl token request

ca83b1d

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Dec 29, 2022

ti-chi-bot requested review from rleungx and Yisaer December 29, 2022 12:14

lhy1024 reviewed Dec 29, 2022

View reviewed changes

CabinfeverB changed the title ~~resource_manager: implement about roken request~~ resource_manager: implement about token request Dec 29, 2022

nolouch changed the title ~~resource_manager: implement about token request~~ resource_manager: implement token assignment in server Dec 30, 2022

nolouch reviewed Jan 3, 2023

View reviewed changes

CabinfeverB added 2 commits January 3, 2023 17:29

impl token request and fix update bug

321c2ad

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

add test

0fc2ca5

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com> impl token request and fix update bug Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot removed the do-not-merge/needs-linked-issue label Jan 3, 2023

glorv reviewed Jan 4, 2023

View reviewed changes

nolouch reviewed Jan 5, 2023

View reviewed changes

CabinfeverB added 2 commits January 5, 2023 16:48

address comment

ef25725

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

address comment

6a13cab

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 6, 2023

CabinfeverB added 2 commits January 6, 2023 17:18

merge master

2d2fc55

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

merge master

887d221

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

ti-chi-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 6, 2023

nolouch mentioned this pull request Jan 6, 2023

Global Resource Quota Control pingcap/tidb#38825

Closed

41 tasks

address comment

d5aa646

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

nolouch reviewed Jan 9, 2023

View reviewed changes

address comment

5fb09c9

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

address comment

fc04c44

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

JmPotato reviewed Jan 10, 2023

View reviewed changes

nolouch approved these changes Jan 10, 2023

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 10, 2023

address comment

3896b95

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

JmPotato approved these changes Jan 10, 2023

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 10, 2023

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jan 10, 2023

Merge branch 'master' into resource-manager/server_part3

3865ab3

ti-chi-bot merged commit 4363701 into tikv:master Jan 10, 2023

JmPotato deleted the resource-manager/server_part3 branch January 11, 2023 02:45

CabinfeverB mentioned this pull request Jan 11, 2023

Tracking issue for resource manager #5851

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

resource_manager: implement token assignment in server #5809

resource_manager: implement token assignment in server #5809

CabinfeverB commented Dec 29, 2022 •

edited

Loading

ti-chi-bot commented Dec 29, 2022 •

edited

Loading

lhy1024 left a comment

codecov bot commented Dec 29, 2022 •

edited

Loading

nolouch Jan 3, 2023

nolouch left a comment

glorv Jan 4, 2023

glorv Jan 4, 2023

nolouch Jan 5, 2023

nolouch Jan 5, 2023

CabinfeverB Jan 5, 2023

nolouch Jan 5, 2023

CabinfeverB Jan 5, 2023

nolouch Jan 5, 2023

CabinfeverB Jan 5, 2023

nolouch Jan 5, 2023

nolouch commented Jan 6, 2023

nolouch Jan 9, 2023 •

edited

Loading

JmPotato Jan 10, 2023

JmPotato Jan 10, 2023

nolouch left a comment

nolouch Jan 10, 2023

JmPotato commented Jan 10, 2023

ti-chi-bot commented Jan 10, 2023

ti-chi-bot commented Jan 10, 2023

ti-chi-bot commented Jan 10, 2023


		// GroupTokenBucket is a token bucket for a resource group.
		// TODO: statistics Consumption

	// TODO: statistics Consumption
	// TODO: statistics consumption @JmPotato

	Consumption *rmpb.TokenBucketsRequest `json:"consumption,omitempty"`
	Consumption *rmpb.Consumption `json:"consumption,omitempty"`

resource_manager: implement token assignment in server #5809

resource_manager: implement token assignment in server #5809

Conversation

CabinfeverB commented Dec 29, 2022 • edited Loading

What problem does this PR solve?

What is changed and how does it work?

Check List

Release note

ti-chi-bot commented Dec 29, 2022 • edited Loading

lhy1024 left a comment

Choose a reason for hiding this comment

codecov bot commented Dec 29, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

nolouch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolouch commented Jan 6, 2023

nolouch Jan 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nolouch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JmPotato commented Jan 10, 2023

ti-chi-bot commented Jan 10, 2023

ti-chi-bot commented Jan 10, 2023

ti-chi-bot commented Jan 10, 2023

CabinfeverB commented Dec 29, 2022 •

edited

Loading

ti-chi-bot commented Dec 29, 2022 •

edited

Loading

codecov bot commented Dec 29, 2022 •

edited

Loading

nolouch Jan 9, 2023 •

edited

Loading