statistics: trigger auto-analyze based on histogram row count #24382

eurekaka · 2021-04-29T10:25:45Z

What problem does this PR solve?

Issue Number: close #24237, related to #26282

Problem Summary:

Trigger auto analyze in a more proper way.

What is changed and how it works?

What's Changed:

Use the histogram row count as the base count for checking auto analyze, instead of the Count in Table.

Related changes

N/A

Check List

Tests

Unit test

Side effects

N/A

Release note

Trigger auto-analyze based on histogram row count

ti-chi-bot · 2021-04-29T10:25:46Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

qw4990
time-and-fate

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

purelind · 2021-05-02T22:53:40Z

/run-all-tests

statistics/handle/update.go

time-and-fate · 2021-05-06T12:37:55Z

statistics/table.go

+func (t *Table) ColHistCount() float64 {
+	for _, col := range t.Columns {
+		if col != nil {


I think we need to check Table.Pseudo here.
Probably Column.IsInvalid() also.

Why do we only try to get row count from columns? I think indexes are also useful.

1, autoAnalyzeTable has already checked Table.Pseudo;
2, We should NOT check Column.IsInvalid(), since our purpose here is to get the row count of last analyze, even if Column.IsInvalid() is true, the row count is what we want;
3, We should NOT get row count from indexes, normally, index count should be same as column count, but if analyze index happens, index count is not what we want;

OK. 👌

There is one special case. When the stats for a column is not loaded and there are NULLs in this column, the (*Column).TotalRowCount() will equal null count, which is incorrect. We need to check against this case.

time-and-fate

I find that the newly added unit test can pass on the current master branch.

statistics/handle/update_test.go

winoros · 2021-06-21T09:18:21Z

statistics/table.go

@@ -214,6 +214,16 @@ func (t *Table) GetStatsInfo(ID int64, isIndex bool) (int64, *Histogram, *CMSket
 	return int64(colStatsInfo.TotalRowCount()), colStatsInfo.Histogram.Copy(), colStatsInfo.CMSketch.Copy(), colStatsInfo.TopN.Copy(), colStatsInfo.FMSketch.Copy()
 }

+// ColHistCount returns the count of the column histograms.
+func (t *Table) ColHistCount() float64 {


The name could be changed since the topn is split out of the histogram.

statistics/handle/update_test.go

qw4990 · 2021-07-28T07:22:21Z

/merge

time-and-fate · 2021-07-28T07:39:10Z

/run-check_dev_2
/run-unit-test

qw4990 · 2021-07-28T08:31:23Z

/merge

time-and-fate · 2021-07-28T08:50:43Z

/run-unit-test

time-and-fate · 2021-07-28T08:56:54Z

/run-unit-test

time-and-fate · 2021-07-28T09:15:28Z

/run-unit-test

time-and-fate · 2021-07-28T09:31:01Z

/run-unit-test

zhouqiang-cl · 2021-07-28T09:54:20Z

/run-unit-test

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-07-28T10:03:56Z

cherry pick to release-4.0 in PR #26706

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-07-28T10:05:25Z

cherry pick to release-5.0 in PR #26707

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-07-28T10:06:26Z

cherry pick to release-5.1 in PR #26708

#26707)

#26706)

#26708)

eurekaka added type/enhancement The issue or PR belongs to an enhancement. sig/planner SIG: Planner component/statistics labels Apr 29, 2021

eurekaka requested a review from a team as a code owner April 29, 2021 10:25

eurekaka requested review from XuHuaiyu and removed request for a team April 29, 2021 10:25

ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 29, 2021

eurekaka requested review from time-and-fate, qw4990 and winoros and removed request for XuHuaiyu April 29, 2021 10:26

time-and-fate reviewed May 6, 2021

View reviewed changes

statistics/handle/update_test.go Outdated Show resolved Hide resolved

statistics/handle/update_test.go Outdated Show resolved Hide resolved

eurekaka force-pushed the auto_analyze branch 2 times, most recently from 690c99c to fa2a4c4 Compare May 8, 2021 10:07

statistics: trigger auto-analyze based on histogram row count

14cb4a7

eurekaka force-pushed the auto_analyze branch from fa2a4c4 to 14cb4a7 Compare May 8, 2021 10:21

eurekaka requested a review from time-and-fate May 8, 2021 10:22

winoros reviewed Jun 21, 2021

View reviewed changes

qw4990 removed their request for review June 24, 2021 06:16

qw4990 self-requested a review July 15, 2021 13:01

qw4990 added needs-cherry-pick-4.0 needs-cherry-pick-release-5.0 needs-cherry-pick-release-5.1 labels Jul 15, 2021

qw4990 added 3 commits July 15, 2021 21:02

Merge branch 'master' into auto_analyze

959aa3d

Merge branch 'master' into auto_analyze

380c353

Merge branch 'master' into auto_analyze

728f7b9

qw4990 reviewed Jul 19, 2021

View reviewed changes

statistics/handle/update_test.go Outdated Show resolved Hide resolved

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jul 28, 2021

time-and-fate and others added 2 commits July 28, 2021 14:52

Merge branch 'master' into auto_analyze

753089a

Merge branch 'master' into auto_analyze

d8e66ad

qw4990 and others added 3 commits July 28, 2021 15:54

Merge branch 'master' into auto_analyze

5ace68e

Merge branch 'master' into auto_analyze

1d83683

Merge branch 'master' into auto_analyze

5472c8f

Merge branch 'master' into auto_analyze

851e136

ti-chi-bot merged commit 50ae2d8 into pingcap:master Jul 28, 2021

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 28, 2021

cherry pick pingcap#24382 to release-4.0

10bec34

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jul 28, 2021

statistics: trigger auto-analyze based on histogram row count (#24382) #26706

Merged

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 28, 2021

cherry pick pingcap#24382 to release-5.0

0c5b701

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jul 28, 2021

statistics: trigger auto-analyze based on histogram row count (#24382) #26707

Merged

ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Jul 28, 2021

cherry pick pingcap#24382 to release-5.1

34b2ae4

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Jul 28, 2021

statistics: trigger auto-analyze based on histogram row count (#24382) #26708

Merged

feitian124 mentioned this pull request Jul 29, 2021

util/decoder: migrate test-infra to testify #26629

Merged

eurekaka deleted the auto_analyze branch August 23, 2021 09:05

ti-chi-bot pushed a commit that referenced this pull request Aug 30, 2021

statistics: trigger auto-analyze based on histogram row count (#24382) (

557be78

#26707)

ti-chi-bot pushed a commit that referenced this pull request Sep 10, 2021

statistics: trigger auto-analyze based on histogram row count (#24382) (

4b8d5cb

#26706)

ti-chi-bot pushed a commit that referenced this pull request Sep 15, 2021

statistics: trigger auto-analyze based on histogram row count (#24382) (

4194fe8

#26708)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

statistics: trigger auto-analyze based on histogram row count #24382

statistics: trigger auto-analyze based on histogram row count #24382

eurekaka commented Apr 29, 2021 •

edited by qw4990

Loading

ti-chi-bot commented Apr 29, 2021 •

edited

Loading

purelind commented May 2, 2021

time-and-fate May 6, 2021

time-and-fate May 6, 2021

eurekaka May 8, 2021

time-and-fate May 10, 2021 •

edited

Loading

time-and-fate left a comment

winoros Jun 21, 2021

qw4990 commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

qw4990 commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

zhouqiang-cl commented Jul 28, 2021

ti-srebot commented Jul 28, 2021

ti-srebot commented Jul 28, 2021

ti-srebot commented Jul 28, 2021

statistics: trigger auto-analyze based on histogram row count #24382

statistics: trigger auto-analyze based on histogram row count #24382

Conversation

eurekaka commented Apr 29, 2021 • edited by qw4990 Loading

What problem does this PR solve?

What is changed and how it works?

Related changes

Check List

Release note

ti-chi-bot commented Apr 29, 2021 • edited Loading

purelind commented May 2, 2021

time-and-fate May 6, 2021

Choose a reason for hiding this comment

time-and-fate May 6, 2021

Choose a reason for hiding this comment

eurekaka May 8, 2021

Choose a reason for hiding this comment

time-and-fate May 10, 2021 • edited Loading

Choose a reason for hiding this comment

time-and-fate left a comment

Choose a reason for hiding this comment

winoros Jun 21, 2021

Choose a reason for hiding this comment

qw4990 commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

qw4990 commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

time-and-fate commented Jul 28, 2021

zhouqiang-cl commented Jul 28, 2021

ti-srebot commented Jul 28, 2021

ti-srebot commented Jul 28, 2021

ti-srebot commented Jul 28, 2021

eurekaka commented Apr 29, 2021 •

edited by qw4990

Loading

ti-chi-bot commented Apr 29, 2021 •

edited

Loading

time-and-fate May 10, 2021 •

edited

Loading