statistics: batch insert topn and buckets to storage to speed up loading stats #36948
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
release-note-none
|
Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/44a4157856825e0d074c488bde74f1e46ea6d89a |
hawkingrei
reviewed
Aug 7, 2022
| @@ -1107,6 +1107,83 @@ func (h *Handle) StatsMetaCountAndModifyCount(tableID int64) (int64, int64, erro | |||
| return count, modifyCount, nil | |||
| } | |||
|
|
|||
| func saveTopNToStorage(ctx context.Context, exec sqlexec.SQLExecutor, tableID int64, isIndex int, histID int64, topN *statistics.TopN) error { | |||
There was a problem hiding this comment.
when you enable bce to build it, you can see the bce case.
go build -gcflags="-d=ssa/check_bce/debug=1" ./statistics/handle/...
statistics/handle/handle.go:1122:92: Found IsInBounds
statistics/handle/handle.go:1122:114: Found IsInBounds
statistics/handle/handle.go:1152:23: Found IsInBounds
statistics/handle/handle.go:1169:118: Found IsInBounds
statistics/handle/handle.go:1169:186: Found IsInBounds
There was a problem hiding this comment.
Can you change the code to avoid some unnecessary bounds checks?
There was a problem hiding this comment.
OK, I will fix it.
wjhuang2016
approved these changes
Aug 9, 2022
hawkingrei
approved these changes
Aug 10, 2022
ti-chi-bot
added
status/LGT2
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: aff13dc
|
ti-chi-bot
added
the
status/can-merge
TiDB MergeCI notify🔴 Bad News! New failing [2] after this pr merged.
|
morgo
added a commit
to morgo/tidb
that referenced
this pull request
Aug 11, 2022
…verage * upstream/master: (98 commits) planner: provide a unified interface to set and get user variables and types (pingcap#37046) *: fix check-2 to bazel (pingcap#37055) distsql: revert 37019 (pingcap#37048) planner: introduce new functions for the general statement in session interface (pingcap#37024) session: make BenchmarkCompileExecutePreparedStmt update to date (pingcap#36960) server: Change proxyprotocol to lazy mode (pingcap#36301) test: fix `TestDDLSchedulingMultiTimes` and `TestConcurrentDDLSwitch` timeout (pingcap#37043) makefile: add bazel realtikvtest (pingcap#37042) distsql: make analyze request SI (pingcap#37019) *: gofmt code for adapting go 1.19 (pingcap#36832) planner: fix data race in the TestInlineProjection4HashJoinIssue15316 (pingcap#37014) planner: introduce Session.CacheGeneralStmt for general plan cache (pingcap#37028) server: revise scheme code (pingcap#37021) planner: fix unstable test TestPushDownSelectionForMPP (pingcap#37038) executor: more precise memory tracking (pingcap#35785) executor: add privilege check for prepare stmt (pingcap#36933) *: optimize temptable.SessionSnapshotInterceptor (pingcap#36999) statistics: batch insert topn and buckets to storage to speed up loading stats (pingcap#36948) planner: remove keyword `prepared` in plan cache backend (pingcap#37023) util: fix unstable test TestCheckCN (pingcap#37016) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #36947
Problem Summary:
What is changed and how it works?
Batch insert topn and buckets to storage to speed up loading stats from JSON files.
Check List
Tests
Test the time of loading stats of several tables.
Before the PR:
After the PR:
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.