statistics: avoid stats meta full load after table analysis #57756

Rustin170506 · 2024-11-27T05:53:31Z

What problem does this PR solve?

Issue Number: close #57631

Problem Summary:

We need to support updating the stats of only a few tables within the update function. This will help reduce the function's duration after the table analysis is completed.

The analysis duration is stable:

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.22 sec)

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.74 sec)

mysql> analyze table sbtest111.sbtest2;
Query OK, 0 rows affected, 10 warnings (6.12 sec)

What changed and how does it work?

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Rustin170506 · 2024-11-27T06:13:38Z

/retest

codecov · 2024-11-27T06:21:03Z

Codecov Report

Attention: Patch coverage is 95.65217% with 3 lines in your changes missing coverage. Please review.

Project coverage is 73.8288%. Comparing base (8fe0618) to head (3808148).
Report is 1 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57756        +/-   ##
================================================
+ Coverage   73.1430%   73.8288%   +0.6858%     
================================================
  Files          1671       1701        +30     
  Lines        460633     468749      +8116     
================================================
+ Hits         336921     346072      +9151     
+ Misses       103006     101245      -1761     
- Partials      20706      21432       +726

Flag	Coverage Δ
integration	`46.1426% <72.4637%> (?)`
unit	`72.2917% <95.6521%> (+0.0113%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`46.0253% <ø> (+0.0331%)`	⬆️

Rustin170506

🔢 Self-check (PR reviewed by myself and ready for feedback.)

Rustin170506 · 2024-11-27T06:42:45Z

pkg/executor/analyze.go

@@ -99,6 +99,14 @@ func (e *AnalyzeExec) Next(ctx context.Context, _ *chunk.Chunk) error {
 	if len(tasks) == 0 {
 		return nil
 	}
+	tableAndPartitionIDs := make([]int64, 0, len(tasks))


Maybe we can get this information from the previous function, but I’d prefer not to introduce extra refactoring in this PR since we need to cherry-pick it to 8.5.

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506 · 2024-11-29T05:56:45Z

/retest

Rustin170506

🔢 Self-check (PR reviewed by myself and ready for feedback.)

Rustin170506 · 2024-11-29T08:13:01Z

/retest

time-and-fate

Others LGTM.

pkg/statistics/handle/storage/read.go

Signed-off-by: Rustin170506 <tech@rustin.me>

time-and-fate

/hold

pkg/statistics/handle/types/interfaces.go

pkg/statistics/handle/storage/update.go

ti-chi-bot · 2024-12-02T14:03:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: time-and-fate, winoros

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [time-and-fate,winoros]
~~pkg/statistics/OWNERS~~ [time-and-fate,winoros]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-12-02T14:03:56Z

[LGTM Timeline notifier]

Timeline:

2024-12-02 06:53:40.061517703 +0000 UTC m=+1051407.681172220: ☑️ agreed by winoros.
2024-12-02 14:03:54.941472793 +0000 UTC m=+1077222.561127308: ☑️ agreed by time-and-fate.

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506

🔢 Self-check (PR reviewed by myself and ready for feedback.)

Rustin170506 · 2024-12-03T05:11:16Z

/unhold

ti-chi-bot · 2024-12-03T05:56:02Z

In response to a cherrypick label: new pull request created to branch release-8.5: #57911.

…57911) close #57631

ti-chi-bot bot added do-not-merge/needs-triage-completed release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed do-not-merge/needs-triage-completed labels Nov 27, 2024

Rustin170506 commented Nov 27, 2024

View reviewed changes

Rustin170506 requested review from winoros and time-and-fate November 27, 2024 07:00

Rustin170506 added 4 commits November 28, 2024 11:35

statistics: avoid stats meta full load after table analysis

e82e4b4

Signed-off-by: Rustin170506 <tech@rustin.me>

fix: make lint happy

e18ec37

Signed-off-by: Rustin170506 <tech@rustin.me>

test: add an unit test

8d6e156

Signed-off-by: Rustin170506 <tech@rustin.me>

docs: add comments

519a0d6

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506 force-pushed the rustin-patch-analysis-full-load branch from 675167d to 519a0d6 Compare November 28, 2024 03:37

Rustin170506 added 2 commits November 28, 2024 13:34

fix: bump to 10000

0376db2

Signed-off-by: Rustin170506 <tech@rustin.me>

debug: print rows

8576b1e

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506 force-pushed the rustin-patch-analysis-full-load branch from fa1f501 to 8576b1e Compare November 28, 2024 11:30

Rustin170506 added 2 commits November 28, 2024 20:40

debug: print delta number

dd2c341

Signed-off-by: Rustin170506 <tech@rustin.me>

fix: remove logs

238a1c4

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506 commented Nov 29, 2024

View reviewed changes

winoros approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 2, 2024

time-and-fate reviewed Dec 2, 2024

View reviewed changes

pkg/statistics/handle/storage/read.go Outdated Show resolved Hide resolved

fix: remove useless change

6febcf8

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506 requested a review from time-and-fate December 2, 2024 13:37

time-and-fate approved these changes Dec 2, 2024

View reviewed changes

pkg/statistics/handle/types/interfaces.go Outdated Show resolved Hide resolved

pkg/statistics/handle/storage/update.go Outdated Show resolved Hide resolved

ti-chi-bot bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 2, 2024

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 2, 2024

ti-chi-bot bot added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 3, 2024

fix: remove unnessary code

b50b414

Signed-off-by: Rustin170506 <tech@rustin.me>

Rustin170506 commented Dec 3, 2024

View reviewed changes

Merge branch 'master' into rustin-patch-analysis-full-load

3808148

ti-chi-bot bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2024

ti-chi-bot bot merged commit f585f5d into pingcap:master Dec 3, 2024
25 checks passed

ti-chi-bot mentioned this pull request Dec 3, 2024

statistics: avoid stats meta full load after table analysis (#57756) #57911

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 3, 2024

statistics: avoid stats meta full load after table analysis (#57756) (#…

6087f99

…57911) close #57631

Rustin170506 mentioned this pull request Dec 19, 2024

nightly : The ANALYZE TABLE command does not return a result, but it is actually successful #54475

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: avoid stats meta full load after table analysis #57756

statistics: avoid stats meta full load after table analysis #57756

Rustin170506 commented Nov 27, 2024 •

edited

Loading

Rustin170506 commented Nov 27, 2024

codecov bot commented Nov 27, 2024 •

edited

Loading

Rustin170506 left a comment

Rustin170506 Nov 27, 2024

Rustin170506 commented Nov 29, 2024

Rustin170506 left a comment

Rustin170506 commented Nov 29, 2024

time-and-fate left a comment

time-and-fate left a comment

ti-chi-bot bot commented Dec 2, 2024

ti-chi-bot bot commented Dec 2, 2024

Rustin170506 left a comment

Rustin170506 commented Dec 3, 2024

ti-chi-bot commented Dec 3, 2024

statistics: avoid stats meta full load after table analysis #57756

statistics: avoid stats meta full load after table analysis #57756

Conversation

Rustin170506 commented Nov 27, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Rustin170506 commented Nov 27, 2024

codecov bot commented Nov 27, 2024 • edited Loading

Codecov Report

Rustin170506 left a comment

Choose a reason for hiding this comment

Rustin170506 Nov 27, 2024

Choose a reason for hiding this comment

Rustin170506 commented Nov 29, 2024

Rustin170506 left a comment

Choose a reason for hiding this comment

Rustin170506 commented Nov 29, 2024

time-and-fate left a comment

Choose a reason for hiding this comment

time-and-fate left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 2, 2024

ti-chi-bot bot commented Dec 2, 2024

[LGTM Timeline notifier]

Rustin170506 left a comment

Choose a reason for hiding this comment

Rustin170506 commented Dec 3, 2024

ti-chi-bot commented Dec 3, 2024

Rustin170506 commented Nov 27, 2024 •

edited

Loading

codecov bot commented Nov 27, 2024 •

edited

Loading