planner, workload-based learning: The setup process framework of workload-based learning #58132

elsa0520 · 2024-12-10T07:28:15Z

What problem does this PR solve?

Issue Number: ref #58131

Problem Summary:

What changed and how does it work?

The setup process framework of workload-based learning.
When tidb setup, the workload-based learnng worker such as read tablecost worker will setup at the same time.
It will start a new go routine to execute workers periodically. All workload-based learning related workers will be started in this function SetupWorkloadBasedLearningWorker"

Two new configurations (TODO set value)

EnableWorkloadBasedLearning: Control whether enable all workload-based analyze workers. Default is false before this feature is ready
WorkloadBasedLearningInterval: Control the time interval between each round of analysis task. Default is 1 days.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
Now, we can only disable it before the feature is not ready. So I manual test the setup worker.
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-12-10T07:28:32Z

Hi @elsa0520. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-12-10T08:57:55Z

Codecov Report

Attention: Patch coverage is 86.79245% with 7 lines in your changes missing coverage. Please review.

Project coverage is 75.2435%. Comparing base (68ac9ec) to head (82ea518).
Report is 71 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #58132        +/-   ##
================================================
+ Coverage   73.1841%   75.2435%   +2.0594%     
================================================
  Files          1675       1724        +49     
  Lines        461917     474491     +12574     
================================================
+ Hits         338050     357024     +18974     
+ Misses       103127      95262      -7865     
- Partials      20740      22205      +1465

Flag	Coverage Δ
integration	`49.2410% <56.6037%> (?)`
unit	`72.6734% <86.7924%> (+0.3586%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.6910% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`61.9710% <ø> (+16.0102%)`	⬆️

ghazalfamilyusa · 2024-12-10T19:52:45Z

Two points:

Do you actually mean that this PR close the corresponding 58131 issue?
Need unit test for the initial functionality.

pkg/domain/domain.go

ghazalfamilyusa · 2024-12-10T20:00:47Z

pkg/domain/domain.go

+}
+
+func (do *Domain) readTableCostWorker(wbLearningHandle *workloadbasedlearning.Handle) {
+	defer util.Recover(metrics.LabelDomain, "readTableCostWorker", nil, false)


Comment as well?

pkg/domain/domain.go

pkg/workloadbasedlearning/handle.go

pkg/domain/domain.go

Rustin170506

Thanks!

Some nits.

Rustin170506 · 2024-12-11T09:03:59Z

pkg/workloadbasedlearning/handle.go

+
+package workloadbasedlearning
+
+/*


In Go, we rarely use /* */ comments. See more at https://go.dev/doc/comment to check the package comment style.

elsa0520 · 2024-12-12T02:12:18Z

Two points:

Do you actually mean that this PR close the corresponding 58131 issue?

Need unit test for the initial functionality.

No, I changed the comment to ref 58131
I will add unit test in the future pr link. Only test the tidb setup is a little bit difficult in unit test

ti-chi-bot · 2024-12-12T03:28:10Z

[LGTM Timeline notifier]

Timeline:

2024-12-10 23:29:11.977138879 +0000 UTC m=+394742.065941417: ✖️🔁 reset by ghazalfamilyusa.
2024-12-11 09:06:36.422704571 +0000 UTC m=+429386.511507310: ☑️ agreed by Rustin170506.
2024-12-12 03:28:09.165968061 +0000 UTC m=+495479.254770604: ☑️ agreed by ghazalfamilyusa.

ti-chi-bot · 2024-12-12T21:09:49Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: benmeadowcroft, ghazalfamilyusa, qw4990, Rustin170506, wjhuang2016

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Rustin170506,qw4990,wjhuang2016]
~~pkg/domain/OWNERS~~ [wjhuang2016]
~~pkg/sessionctx/variable/OWNERS~~ [benmeadowcroft]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

elsa0520 · 2024-12-13T02:13:44Z

/test unit-test

tiprow · 2024-12-13T02:14:05Z

@elsa0520: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

qw4990 · 2024-12-13T05:57:33Z

/retest

tiprow · 2024-12-13T05:57:53Z

@qw4990: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

elsa0520 · 2024-12-15T23:16:03Z

/test pull-br-integration-test

tiprow · 2024-12-15T23:16:25Z

@elsa0520: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-br-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 10, 2024

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 10, 2024

elsa0520 added 7 commits December 10, 2024 16:32

The setup process of Workload-based Learning worker

c394f46

add comment1

d24429e

add comment

16fe756

fix ci

996e81c

fix ci

effb158

fix ci

f6712f0

fix linter

599cac6

elsa0520 force-pushed the learning-tablecost branch from b062e0e to 599cac6 Compare December 10, 2024 08:38

ghazalfamilyusa suggested changes Dec 10, 2024

View reviewed changes

qw4990 reviewed Dec 11, 2024

View reviewed changes

pkg/domain/domain.go Show resolved Hide resolved

qw4990 reviewed Dec 11, 2024

View reviewed changes

pkg/workloadbasedlearning/handle.go Show resolved Hide resolved

qw4990 reviewed Dec 11, 2024

View reviewed changes

pkg/domain/domain.go Show resolved Hide resolved

add comment

d0b0e00

Rustin170506 approved these changes Dec 11, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 11, 2024

elsa0520 added 2 commits December 12, 2024 10:21

change comment

d03c8c1

fix

a741246

ghazalfamilyusa approved these changes Dec 12, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 12, 2024

fix ci

82ea518

qw4990 approved these changes Dec 12, 2024

View reviewed changes

wjhuang2016 approved these changes Dec 12, 2024

View reviewed changes

benmeadowcroft approved these changes Dec 12, 2024

View reviewed changes

ti-chi-bot bot added the approved label Dec 12, 2024

ti-chi-bot bot merged commit 59dff48 into pingcap:master Dec 16, 2024
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner, workload-based learning: The setup process framework of workload-based learning #58132

planner, workload-based learning: The setup process framework of workload-based learning #58132

elsa0520 commented Dec 10, 2024 •

edited

Loading

tiprow bot commented Dec 10, 2024

codecov bot commented Dec 10, 2024 •

edited

Loading

ghazalfamilyusa commented Dec 10, 2024 •

edited

Loading

ghazalfamilyusa Dec 10, 2024

elsa0520 Dec 12, 2024

Rustin170506 left a comment

Rustin170506 Dec 11, 2024

elsa0520 Dec 12, 2024

elsa0520 commented Dec 12, 2024

ti-chi-bot bot commented Dec 12, 2024

ti-chi-bot bot commented Dec 12, 2024

elsa0520 commented Dec 13, 2024

tiprow bot commented Dec 13, 2024

qw4990 commented Dec 13, 2024

tiprow bot commented Dec 13, 2024

elsa0520 commented Dec 15, 2024

tiprow bot commented Dec 15, 2024

planner, workload-based learning: The setup process framework of workload-based learning #58132

planner, workload-based learning: The setup process framework of workload-based learning #58132

Conversation

elsa0520 commented Dec 10, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Dec 10, 2024

codecov bot commented Dec 10, 2024 • edited Loading

Codecov Report

ghazalfamilyusa commented Dec 10, 2024 • edited Loading

ghazalfamilyusa Dec 10, 2024

Choose a reason for hiding this comment

elsa0520 Dec 12, 2024

Choose a reason for hiding this comment

Rustin170506 left a comment

Choose a reason for hiding this comment

Rustin170506 Dec 11, 2024

Choose a reason for hiding this comment

elsa0520 Dec 12, 2024

Choose a reason for hiding this comment

elsa0520 commented Dec 12, 2024

ti-chi-bot bot commented Dec 12, 2024

[LGTM Timeline notifier]

ti-chi-bot bot commented Dec 12, 2024

elsa0520 commented Dec 13, 2024

tiprow bot commented Dec 13, 2024

qw4990 commented Dec 13, 2024

tiprow bot commented Dec 13, 2024

elsa0520 commented Dec 15, 2024

tiprow bot commented Dec 15, 2024

elsa0520 commented Dec 10, 2024 •

edited

Loading

codecov bot commented Dec 10, 2024 •

edited

Loading

ghazalfamilyusa commented Dec 10, 2024 •

edited

Loading