Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor, infoschema: Add cluster_statements_summary_evicted table to TiDB #25418

Merged
merged 138 commits into from
Jun 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
1a96cbb
util/kvcache: enhance LRU Cache
ClSlaid Apr 23, 2021
0c8920a
Merge branch 'master' into master
ClSlaid Apr 23, 2021
25ae806
Merge branch 'master' into master
ClSlaid Apr 23, 2021
529e346
Util/kvcache: Typo fix
ClSlaid Apr 23, 2021
38fa019
util/kvcache/simple_lru.go: typo fix
ClSlaid Apr 25, 2021
d78ca46
util/kvcache: fix test
ClSlaid Apr 25, 2021
1cddd8f
Merge branch 'master' into master
crazycs520 Apr 25, 2021
33d865f
util/kvcache: synced onEvict func -> asynced
ClSlaid Apr 25, 2021
2d75e37
Merge branch 'master' of github.com:ClSlaid/tidb
ClSlaid Apr 25, 2021
05703d8
util/kvcache: formatted
ClSlaid Apr 25, 2021
eaf1caa
Merge branch 'master' into master
crazycs520 Apr 25, 2021
0163613
Merge branch 'master' into master
ti-chi-bot Apr 25, 2021
0935f75
Merge branch 'master' into master
ti-chi-bot Apr 25, 2021
c0227b6
Statements Summary Evicted Prototype Commit
ClSlaid Apr 30, 2021
0a168fe
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid Apr 30, 2021
b447b86
Fix STATEMENTS_SUMMARY_EVICTED
ClSlaid May 8, 2021
8e8792b
util/stmtsummary: Fix evicted.go
ClSlaid May 10, 2021
2a8ed11
Merge branch 'pingcap:master' into master
ClSlaid May 10, 2021
12b6a87
Merge branch 'master' into master
ClSlaid May 10, 2021
373d344
util/stmtsummary: Delete debug code
ClSlaid May 10, 2021
0f24624
util/stmtsummary: add test to EVICTED_COUNT
ClSlaid May 11, 2021
e2a18b1
util/stmtsummary: Add test to evicted count
ClSlaid May 12, 2021
6b7da7e
go.sum: disable fail-point
ClSlaid May 12, 2021
ebad974
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 12, 2021
3fcd196
fix merge problems
ClSlaid May 12, 2021
073ff06
clean up evicted count
ClSlaid May 14, 2021
32e8df9
typo fix
ClSlaid May 14, 2021
1092e72
performance improve and typo fix
ClSlaid May 14, 2021
7e8a8d8
simplify logic in addEvicted
ClSlaid May 17, 2021
252b43a
beautify evicted.go && typo fix
ClSlaid May 17, 2021
015b145
fix nil pointer bug in evicted.go
ClSlaid May 19, 2021
e061027
fix zero quota test for kvcache
ClSlaid May 20, 2021
d3bf818
Add test to evicted.go and some bug fixes
ClSlaid May 20, 2021
f59155b
typo fix in executor and infoschema
ClSlaid May 20, 2021
efcad35
typo fix
ClSlaid May 21, 2021
80167a3
try fix git merge problem
ClSlaid May 21, 2021
dbc94c7
Add more test to evicted count
ClSlaid May 21, 2021
756c53e
evicted test full cover
ClSlaid May 24, 2021
43a4adc
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 24, 2021
be4b4e4
fix merge conflict
ClSlaid May 24, 2021
c6ce248
fix nil pointer problem
ClSlaid May 24, 2021
a8be817
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 24, 2021
fdec81a
test refactoring
ClSlaid May 24, 2021
01a92a4
format and add license.
ClSlaid May 24, 2021
9c503e5
fix empty table error and clean up useless codes.
ClSlaid May 25, 2021
b2278b2
Add test to table and more test to evicted count
ClSlaid May 25, 2021
d3ea973
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 25, 2021
a3ed839
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 26, 2021
9b243a5
typo fix
ClSlaid May 26, 2021
eaffa1c
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 26, 2021
413ed0d
make check
ClSlaid May 26, 2021
c91e5e3
OUTDATED AGAIN???: Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 26, 2021
0bd1f5e
fix go.sum
ClSlaid May 26, 2021
f282833
try fix data racing
ClSlaid May 26, 2021
16c83f9
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 26, 2021
cca919b
try fix data racing.
ClSlaid May 26, 2021
52cb27b
try fix data racing again.
ClSlaid May 26, 2021
a31059d
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 26, 2021
4c9e65f
Merge branch 'master' into master
crazycs520 May 27, 2021
b7a87e1
try fix data racing again again.
ClSlaid May 27, 2021
5725827
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 27, 2021
b0776f5
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 27, 2021
b50b213
revoke change in unrelated files.
ClSlaid May 27, 2021
3b9fa88
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 27, 2021
cafa16a
Merge branch 'master' into master
crazycs520 May 28, 2021
f880324
change interval in tables_test.go
ClSlaid May 28, 2021
f740b34
Merge branch 'master' of github.com:pingcap/tidb
ClSlaid May 28, 2021
6c69920
Merge branch 'master' into master
ti-chi-bot May 28, 2021
d8b32ad
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 1, 2021
4629224
add `other` record to CurrentDatum and HistoryDatum
ClSlaid Jun 1, 2021
184b359
add more test
ClSlaid Jun 2, 2021
39655fc
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 2, 2021
51a56b1
add clean-up in test.
ClSlaid Jun 2, 2021
4d633c4
Merge branch 'master' into fire
ClSlaid Jun 2, 2021
528db85
Merge branch 'master' into fire
ClSlaid Jun 2, 2021
9f2783f
typo fixes.
ClSlaid Jun 4, 2021
7500c7c
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 4, 2021
af77055
typo fix...
ClSlaid Jun 4, 2021
3b13b59
Merge branch 'master' into fire
ClSlaid Jun 7, 2021
6af7eed
Merge branch 'master' into fire
ClSlaid Jun 7, 2021
2935d60
fix CommitBackoffTime in statement_summary_test
ClSlaid Jun 7, 2021
1b10912
Merge branch 'master' into fire
ClSlaid Jun 7, 2021
3bfd7e1
fix duplicated code and typo fix
ClSlaid Jun 7, 2021
53423a3
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 7, 2021
322d938
improve legibility && fix typo error.😩
ClSlaid Jun 7, 2021
0802bd5
typo fix
ClSlaid Jun 7, 2021
f516fd3
move 2 functions to test.go
ClSlaid Jun 7, 2021
03e7d21
add test for addInfo()
ClSlaid Jun 8, 2021
a0601b6
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 8, 2021
5101038
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 8, 2021
3952c98
add integrated test
ClSlaid Jun 8, 2021
e9779fa
try fix lock problem
ClSlaid Jun 8, 2021
d0b4fde
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 8, 2021
27ee905
clean side effects in tables_test.go
ClSlaid Jun 8, 2021
2faf184
change SCHEMA_NAME and DIGEST to NULL
ClSlaid Jun 8, 2021
2ea3d3f
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 8, 2021
fc01d92
typo fix
ClSlaid Jun 8, 2021
b5a3aac
typo fix
ClSlaid Jun 8, 2021
f71f3ab
Merge branch 'master' into fire
ClSlaid Jun 8, 2021
49985cd
Merge branch 'master' into fire
ClSlaid Jun 8, 2021
0f920b2
typo fix...
ClSlaid Jun 8, 2021
c19b20a
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 8, 2021
fbb9f79
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
b95f833
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
4de43c7
typo fix
ClSlaid Jun 9, 2021
6544e56
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
9eccd82
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
4936831
shorten lock range
ClSlaid Jun 9, 2021
b823bb7
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 9, 2021
7044744
remove flag in stmtSummaryDigest
ClSlaid Jun 9, 2021
4aefe6f
fix append nil
ClSlaid Jun 9, 2021
9889df9
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
46620d1
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
ab0cbd8
mutex issues :(
ClSlaid Jun 9, 2021
5824872
Merge branch 'master' into fire
ClSlaid Jun 9, 2021
39b1f06
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 9, 2021
337161b
fix firstSeen
ClSlaid Jun 10, 2021
39daceb
fmt
ClSlaid Jun 10, 2021
8f0b757
Merge branch 'master' into fire
ClSlaid Jun 10, 2021
ce03218
Merge branch 'master' into fire
ClSlaid Jun 10, 2021
de53695
Merge branch 'master' into fire
ti-chi-bot Jun 11, 2021
32787d4
Merge branch 'master' into fire
ti-chi-bot Jun 11, 2021
e7b8aff
Merge branch 'master' into fire
ti-chi-bot Jun 11, 2021
67c3452
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 15, 2021
32d9669
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 15, 2021
143ed54
add ClusterTableEvictedCount to TiDB
ClSlaid Jun 15, 2021
6f31c18
Merge branch 'master' into fire
ClSlaid Jun 15, 2021
ba3124e
Merge branch 'master' into fire
ClSlaid Jun 15, 2021
1f3f387
Merge branch 'master' into fire
ClSlaid Jun 15, 2021
a73df4b
Refacted way to set data for statements summary evicted
ClSlaid Jun 15, 2021
684c776
Merge branch 'fire' of github.com:ClSlaid/tidb into fire
ClSlaid Jun 15, 2021
693ec78
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 15, 2021
36fb81e
fix test
ClSlaid Jun 15, 2021
7b8b8d2
Merge branch 'master' of github.com:pingcap/tidb into fire
ClSlaid Jun 15, 2021
e7e61a4
Merge branch 'master' into fire
ClSlaid Jun 17, 2021
cff70d7
Merge branch 'master' into fire
crazycs520 Jun 17, 2021
83e975f
Merge branch 'master' into fire
ti-chi-bot Jun 17, 2021
9c8645c
Merge branch 'master' into fire
ti-chi-bot Jun 17, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions executor/builder.go
Original file line number Diff line number Diff line change
Expand Up @@ -1563,6 +1563,7 @@ func (b *executorBuilder) buildMemTable(v *plannercore.PhysicalMemTable) Executo
strings.ToLower(infoschema.TableStatementsSummaryEvicted),
strings.ToLower(infoschema.ClusterTableStatementsSummary),
strings.ToLower(infoschema.ClusterTableStatementsSummaryHistory),
strings.ToLower(infoschema.ClusterTableStatementsSummaryEvicted),
strings.ToLower(infoschema.TablePlacementPolicy),
strings.ToLower(infoschema.TableClientErrorsSummaryGlobal),
strings.ToLower(infoschema.TableClientErrorsSummaryByUser),
Expand Down
16 changes: 8 additions & 8 deletions executor/infoschema_reader.go
Original file line number Diff line number Diff line change
Expand Up @@ -145,11 +145,11 @@ func (e *memtableRetriever) retrieve(ctx context.Context, sctx sessionctx.Contex
err = e.dataForTiKVStoreStatus(sctx)
case infoschema.TableStatementsSummary,
infoschema.TableStatementsSummaryHistory,
infoschema.TableStatementsSummaryEvicted,
infoschema.ClusterTableStatementsSummary,
infoschema.ClusterTableStatementsSummaryHistory:
infoschema.ClusterTableStatementsSummaryHistory,
infoschema.ClusterTableStatementsSummaryEvicted:
err = e.setDataForStatementsSummary(sctx, e.table.Name.O)
case infoschema.TableStatementsSummaryEvicted:
e.setDataForStatementsSummaryEvicted(sctx)
case infoschema.TablePlacementPolicy:
err = e.setDataForPlacementPolicy(sctx)
case infoschema.TableClientErrorsSummaryGlobal,
Expand Down Expand Up @@ -1912,10 +1912,14 @@ func (e *memtableRetriever) setDataForStatementsSummary(ctx sessionctx.Context,
case infoschema.TableStatementsSummaryHistory,
infoschema.ClusterTableStatementsSummaryHistory:
e.rows = stmtsummary.StmtSummaryByDigestMap.ToHistoryDatum(user, isSuper)
case infoschema.TableStatementsSummaryEvicted,
infoschema.ClusterTableStatementsSummaryEvicted:
e.rows = stmtsummary.StmtSummaryByDigestMap.ToEvictedCountDatum()
}
switch tableName {
case infoschema.ClusterTableStatementsSummary,
infoschema.ClusterTableStatementsSummaryHistory:
infoschema.ClusterTableStatementsSummaryHistory,
infoschema.ClusterTableStatementsSummaryEvicted:
rows, err := infoschema.AppendHostInfoToRows(ctx, e.rows)
if err != nil {
return err
Expand Down Expand Up @@ -2101,10 +2105,6 @@ func (e *memtableRetriever) setDataForClusterDeadlock(ctx sessionctx.Context) er
return nil
}

func (e *memtableRetriever) setDataForStatementsSummaryEvicted(ctx sessionctx.Context) {
e.rows = stmtsummary.StmtSummaryByDigestMap.ToEvictedCountDatum()
}

type hugeMemTableRetriever struct {
dummyCloser
table *model.TableInfo
Expand Down
3 changes: 3 additions & 0 deletions infoschema/cluster.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ const (
ClusterTableStatementsSummary = "CLUSTER_STATEMENTS_SUMMARY"
// ClusterTableStatementsSummaryHistory is the string constant of cluster statement summary history table.
ClusterTableStatementsSummaryHistory = "CLUSTER_STATEMENTS_SUMMARY_HISTORY"
// ClusterTableStatementsSummaryEvicted is the string constant of cluster statement summary evict table.
ClusterTableStatementsSummaryEvicted = "CLUSTER_STATEMENTS_SUMMARY_EVICTED"
// ClusterTableTiDBTrx is the string constant of cluster transaction running table.
ClusterTableTiDBTrx = "CLUSTER_TIDB_TRX"
// ClusterTableDeadlocks is the string constant of cluster dead lock table.
Expand All @@ -49,6 +51,7 @@ var memTableToClusterTables = map[string]string{
TableProcesslist: ClusterTableProcesslist,
TableStatementsSummary: ClusterTableStatementsSummary,
TableStatementsSummaryHistory: ClusterTableStatementsSummaryHistory,
TableStatementsSummaryEvicted: ClusterTableStatementsSummaryEvicted,
TableTiDBTrx: ClusterTableTiDBTrx,
TableDeadlocks: ClusterTableDeadlocks,
}
Expand Down
1 change: 1 addition & 0 deletions infoschema/tables.go
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,7 @@ var tableIDMap = map[string]int64{
ClusterTableDeadlocks: autoid.InformationSchemaDBID + 73,
TableDataLockWaits: autoid.InformationSchemaDBID + 74,
TableStatementsSummaryEvicted: autoid.InformationSchemaDBID + 75,
ClusterTableStatementsSummaryEvicted: autoid.InformationSchemaDBID + 76,
}

type columnInfo struct {
Expand Down
34 changes: 32 additions & 2 deletions infoschema/tables_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1374,12 +1374,18 @@ func (s *testTableSuite) TestStmtSummarySensitiveQuery(c *C) {
))
}

// test stmtSummaryEvictedCount
func (s *testTableSuite) TestSimpleStmtSummaryEvictedCount(c *C) {
now := time.Now().Unix()
interval := int64(1800)
beginTimeForCurInterval := now - now%interval
tk := s.newTestKitWithPlanCache(c)
tk.MustExec(fmt.Sprintf("set global tidb_stmt_summary_refresh_interval = %v", interval))

// clean up side effects
defer tk.MustExec("set global tidb_stmt_summary_max_stmt_count = 100")
defer tk.MustExec("set global tidb_stmt_summary_refresh_interval = 1800")

tk.MustExec("set global tidb_enable_stmt_summary = 0")
tk.MustExec("set global tidb_enable_stmt_summary = 1")
// first sql
Expand All @@ -1395,10 +1401,34 @@ func (s *testTableSuite) TestSimpleStmtSummaryEvictedCount(c *C) {
int64(2)),
))
// TODO: Add more tests.
}

// test stmtSummaryEvictedCount cluster table
func (s *testClusterTableSuite) TestStmtSummaryEvictedCountTable(c *C) {
tk := s.newTestKitWithRoot(c)
// disable refreshing
tk.MustExec("set global tidb_stmt_summary_refresh_interval=9999")
// set information_schema.statements_summary's size to 1
tk.MustExec("set global tidb_stmt_summary_max_stmt_count = 1")
// clean up side effects
tk.MustExec("set global tidb_stmt_summary_max_stmt_count = 100")
tk.MustExec("set global tidb_stmt_summary_refresh_interval = 1800")
defer tk.MustExec("set global tidb_stmt_summary_max_stmt_count = 100")
defer tk.MustExec("set global tidb_stmt_summary_refresh_interval = 1800")
// clear information_schema.statements_summary
tk.MustExec("set global tidb_enable_stmt_summary=0")
tk.MustExec("set global tidb_enable_stmt_summary=1")

// make a new session for test...
tk = s.newTestKitWithRoot(c)
// first sql
tk.MustExec("show databases;")
// second sql, evict former sql from stmt_summary
tk.MustQuery("select evicted_count from information_schema.cluster_statements_summary_evicted;").
Check(testkit.Rows("1"))
// after executed the sql above
tk.MustQuery("select evicted_count from information_schema.cluster_statements_summary_evicted;").
Check(testkit.Rows("2"))
// TODO: Add more tests.

}

func (s *testTableSuite) TestStmtSummaryTableOther(c *C) {
Expand Down