disttask: init capacity and check concurrency using cpu count of managed node #49875

D3Hunter · 2023-12-28T06:24:19Z

What problem does this PR solve?

Issue Number: ref #49008

Problem Summary:

What changed and how does it work?

add cpu_count to mysql.dist_framework_meta(reusing existing ver 180)
use cpu-count of managed node as capacity when control resource and checking when submitting new task

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
create new and upgrade cluster, make sure new schema works
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2023-12-28T06:24:36Z

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ywqzzy · 2023-12-28T06:36:03Z

pkg/disttask/framework/scheduler/nodes.go

+	for _, node := range newNodes {
+		nodeIDs = append(nodeIDs, node.ID)
+		if node.CPUCount > 0 {
+			cpuCount = node.CPUCount
+		}
 	}


The cpuCount will be set equal to the last node, is it expected?

we assume them isomorphic, first/last makes no difference

can add a comment~

ywqzzy · 2023-12-28T06:37:33Z

pkg/executor/set.go

+			taskMgr, err := storage.GetTaskManager()
+			if err != nil {
+				return err
+			}
+			return taskMgr.StartManagerSession(ctx, e.Ctx(), serverID, valStr)


If the sysvar is not set, the node won't be used for disttask.
Is it exepected?

why? i just replace it from explicit sql to task_table api

Okay, I missunderstood the code

codecov · 2023-12-28T06:42:52Z

Codecov Report

Merging #49875 (60d81f3) into master (d6aeaf0) will decrease coverage by 0.9114%.
Report is 8 commits behind head on master.
The diff coverage is 64.9122%.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #49875        +/-   ##
================================================
- Coverage   72.0575%   71.1462%   -0.9114%     
================================================
  Files          1437       1433         -4     
  Lines        345628     417141     +71513     
================================================
+ Hits         249051     296780     +47729     
- Misses        76506     101578     +25072     
+ Partials      20071      18783      -1288

Flag	Coverage Δ
integration	`43.9051% <64.9122%> (?)`
unit	`70.9427% <ø> (-1.1148%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9663% <ø> (-2.3466%)`	⬇️
parser	`∅ <ø> (∅)`
br	`46.8935% <ø> (-4.5056%)`	⬇️

ywqzzy · 2023-12-28T06:42:10Z

pkg/disttask/framework/storage/task_table_test.go

@@ -37,6 +39,16 @@ func TestMain(m *testing.M) {
 	goleak.VerifyTestMain(m, opts...)
 }

+func GetCPUCountOfManagedNodes(ctx context.Context, taskMgr *TaskManager) (int, error) {


move to util?

let's do it when it's needed

ywqzzy · 2023-12-28T06:44:48Z

pkg/executor/set.go

+			taskMgr, err := storage.GetTaskManager()
+			if err != nil {
+				return err
+			}
+			return taskMgr.StartManagerSession(ctx, e.Ctx(), serverID, valStr)


Okay, I missunderstood the code

ywqzzy

Rest LGTM

ywqzzy · 2023-12-28T06:46:43Z

pkg/disttask/framework/scheduler/slots.go

-	// TODO: we assume that all nodes managed by dist framework are isomorphic,
-	// but dist owner might run on normal node where the capacity might not be
-	// able to run any task.


Add some comments of isomorphic

see comments in slotManager

pkg/disttask/framework/storage/task_table.go

…ields

okJiang · 2023-12-29T08:04:52Z

pkg/disttask/framework/storage/task_table.go

+	rs, err := ExecSQL(ctx, se, `
+		select host, role, cpu_count
+		from mysql.dist_framework_meta
+		where role = 'background' or role = ''


how about removing this?

ti-chi-bot · 2023-12-29T08:34:02Z

[LGTM Timeline notifier]

Timeline:

2023-12-28 06:55:20.50061185 +0000 UTC m=+1721611.537838779: ☑️ agreed by ywqzzy.
2023-12-29 08:34:01.106464083 +0000 UTC m=+1813932.143691009: ☑️ agreed by okJiang.

…ields

D3Hunter · 2023-12-29T10:51:50Z

/approve

D3Hunter · 2023-12-29T14:47:44Z

/hold

ci fail

bb7133

LGTM

D3Hunter · 2024-01-02T03:09:41Z

ci fail issue, will fix it later, #49950

D3Hunter · 2024-01-02T03:09:45Z

/unhold

D3Hunter · 2024-01-02T03:11:23Z

/retest

tiprow · 2024-01-02T03:11:46Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yudongusa

LGTM for now, but @Benjamin2037 please consider the version rollback design.

ti-chi-bot · 2024-01-02T04:03:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bb7133, D3Hunter, okJiang, yudongusa, ywqzzy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,bb7133,okJiang]
~~pkg/ddl/OWNERS~~ [ywqzzy]
~~pkg/disttask/OWNERS~~ [D3Hunter,ywqzzy]
~~pkg/executor/importer/OWNERS~~ [D3Hunter]
~~pkg/session/OWNERS~~ [yudongusa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2024-02-07T02:33:23Z

In response to a cherrypick label: new pull request created to branch release-7.5: #51021.

D3Hunter added 3 commits December 27, 2023 19:27

change

dffd3f5

comment

64cbdac

test

b927b56

ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 28, 2023

ti-chi-bot bot removed the do-not-merge/needs-linked-issue label Dec 28, 2023

D3Hunter mentioned this pull request Dec 28, 2023

resource control inside dist framework #49008

Open

43 tasks

lint

9b1edd0

ywqzzy reviewed Dec 28, 2023

View reviewed changes

ywqzzy approved these changes Dec 28, 2023

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 28, 2023

D3Hunter added 2 commits December 28, 2023 15:12

fix comments

6532be7

Merge remote-tracking branch 'upstream/master' into add-server-info-f…

10079f7

…ields

okJiang approved these changes Dec 29, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 29, 2023

D3Hunter added 2 commits December 29, 2023 18:46

fix comment

9c2c2e1

Merge remote-tracking branch 'upstream/master' into add-server-info-f…

e983841

…ields

fix extra file

60d81f3

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 29, 2023

bb7133 approved these changes Dec 29, 2023

View reviewed changes

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2024

yudongusa approved these changes Jan 2, 2024

View reviewed changes

ti-chi-bot bot added the approved label Jan 2, 2024

ti-chi-bot bot merged commit 3d939d4 into pingcap:master Jan 2, 2024
17 checks passed

D3Hunter deleted the add-server-info-fields branch January 2, 2024 05:44

This was referenced Jan 3, 2024

cancel import into task failed with error “Error 1105 (HY000): task manager is not initialized” and cancel task stuck when cancel again #49993

Closed

disttask: init node meta inside domain #49996

Merged

jiyfhust mentioned this pull request Jan 7, 2024

[planner:1054]Unknown column 'cpu_count' in 'field list' #50147

Closed

ti-chi-bot added the needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. label Feb 7, 2024

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Feb 7, 2024

This is an automated cherry-pick of pingcap#49875

19f7f1c

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Feb 7, 2024

disttask: init capacity and check concurrency using cpu count of managed node (#49875) #51021

Closed

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disttask: init capacity and check concurrency using cpu count of managed node #49875

disttask: init capacity and check concurrency using cpu count of managed node #49875

D3Hunter commented Dec 28, 2023 •

edited

Loading

tiprow bot commented Dec 28, 2023

ywqzzy Dec 28, 2023

D3Hunter Dec 28, 2023

okJiang Dec 29, 2023

ywqzzy Dec 28, 2023

D3Hunter Dec 28, 2023

ywqzzy Dec 28, 2023

codecov bot commented Dec 28, 2023 •

edited

Loading

ywqzzy Dec 28, 2023

D3Hunter Dec 28, 2023

ywqzzy Dec 28, 2023

ywqzzy left a comment

ywqzzy Dec 28, 2023

D3Hunter Dec 28, 2023

okJiang Dec 29, 2023

ti-chi-bot bot commented Dec 29, 2023

D3Hunter commented Dec 29, 2023

D3Hunter commented Dec 29, 2023

bb7133 left a comment

D3Hunter commented Jan 2, 2024

D3Hunter commented Jan 2, 2024

D3Hunter commented Jan 2, 2024

tiprow bot commented Jan 2, 2024

yudongusa left a comment

ti-chi-bot bot commented Jan 2, 2024

ti-chi-bot commented Feb 7, 2024

disttask: init capacity and check concurrency using cpu count of managed node #49875

disttask: init capacity and check concurrency using cpu count of managed node #49875

Conversation

D3Hunter commented Dec 28, 2023 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Dec 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 28, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ywqzzy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 29, 2023

[LGTM Timeline notifier]

D3Hunter commented Dec 29, 2023

D3Hunter commented Dec 29, 2023

bb7133 left a comment

Choose a reason for hiding this comment

D3Hunter commented Jan 2, 2024

D3Hunter commented Jan 2, 2024

D3Hunter commented Jan 2, 2024

tiprow bot commented Jan 2, 2024

yudongusa left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jan 2, 2024

ti-chi-bot commented Feb 7, 2024

D3Hunter commented Dec 28, 2023 •

edited

Loading

codecov bot commented Dec 28, 2023 •

edited

Loading