planner: use TableInfo.DBID to locate schema #57785

tangenta · 2024-11-28T07:27:19Z

What problem does this PR solve?

Issue Number: close #57783, close #57779

Problem Summary:

When tableInfo is located, it is unnecessary to read all databases to match.

What changed and how does it work?

This PR changes to use TableInfo.DBID to locate the schema.

Check List

Tests

Unit test
Integration test

Manual test (add detailed scripts or steps below)
Use the same environment mentioned in It takes too long time for add index in million tables #57783:

mysql> set @@tidb_mem_quota_query = 10*1024*1024*1024;
Query OK, 0 rows affected (0.22 sec)

mysql> select TABLE_SCHEMA, table_name from INFORMATION_SCHEMA.tables where tidb_table_id=1000470;
+--------------+------------+
| TABLE_SCHEMA | table_name |
+--------------+------------+
| sbtest99     | sbtest2    |
+--------------+------------+
1 row in set (0.05 sec)

mysql> select count(1) from sbtest99.sbtest2;
+----------+
| count(1) |
+----------+
|    20000 |
+----------+
1 row in set (0.08 sec)

mysql> alter table sbtest99.sbtest2 add index idx(ac);
Query OK, 0 rows affected (3.06 sec)

After this PR, it takes

0.05s to query table that filtered with tidb_table_id.
3.06s to add index for a table with 20k rows.

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-11-28T07:27:38Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-11-28T16:27:54Z

Codecov Report

Attention: Patch coverage is 84.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 73.4878%. Comparing base (0e8c23f) to head (7dedf20).
Report is 6 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #57785        +/-   ##
================================================
+ Coverage   72.8945%   73.4878%   +0.5933%     
================================================
  Files          1677       1677                
  Lines        464463     465983      +1520     
================================================
+ Hits         338568     342441      +3873     
+ Misses       104989     102665      -2324     
+ Partials      20906      20877        -29

Flag	Coverage Δ
integration	`43.3750% <80.0000%> (?)`
unit	`72.2743% <84.0000%> (-0.0409%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7673% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`45.7257% <ø> (+0.0120%)`	⬆️

joechenrh · 2024-11-29T02:35:09Z

pkg/planner/core/memtable_infoschema_extractor.go

 	tableSlice []*model.TableInfo,
 ) ([]pmodel.CIStr, []*model.TableInfo, error) {
 	schemaSlice := make([]pmodel.CIStr, 0, len(tableSlice))
 	for i, tbl := range tableSlice {
+		dbInfo, ok := is.SchemaByID(tbl.DBID)


~~Seem like DBID field is not persisted, can we use this field to do search?~~

Sorry, I found it's set in GetTable

tangenta · 2024-11-29T07:17:48Z

pkg/planner/core/memtable_infoschema_extractor.go

+func (e *InfoSchemaBaseExtractor) listExtractedSchemas(
+	is infoschema.InfoSchema,
+) (schemas []pmodel.CIStr, unspecified bool) {


Although we can use schemas == nil to indicate no predicate is specified, developers may get confused to distinguish from len(schemas) == 0.

pkg/planner/core/memtable_infoschema_extractor.go

joechenrh · 2024-11-29T07:47:08Z

pkg/planner/core/memtable_infoschema_extractor.go

@@ -150,7 +159,8 @@ func (e *InfoSchemaBaseExtractor) ListSchemasAndTables(
 			findTablesByID(is, tableIDs, tableNames, tableMap)
 			tableSlice := maps.Values(tableMap)
 			tableSlice = filterSchemaObjectByRegexp(e, ec.table, tableSlice, extractStrTableInfo)
-			return findSchemasForTables(ctx, is, schemas, tableSlice)
+			schemas, unspecified := e.listExtractedSchemas(is)


By the way, it's a bit weird to check predicates by ec.tableID != "" (Not related to this PR although)

Do you mean we should decide this at the compile time? Do you have any suggestions?

Nevermind, because I have no suggestion yet :(

pkg/planner/core/memtable_infoschema_extractor.go

ti-chi-bot · 2024-12-02T02:23:34Z

@joechenrh: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tiancaiamao · 2024-12-02T02:28:08Z

pkg/planner/core/memtable_infoschema_extractor.go

 	schemaSlice := make([]pmodel.CIStr, 0, len(tableSlice))
 	for i, tbl := range tableSlice {
+		dbInfo, ok := is.SchemaByID(tbl.DBID)
+		if !ok {
+			continue


Something is wrong? how about add some log in this situation?

ti-chi-bot · 2024-12-02T04:43:35Z

[LGTM Timeline notifier]

Timeline:

2024-12-02 02:33:23.794338557 +0000 UTC m=+1035791.413993091: ☑️ agreed by tiancaiamao.
2024-12-02 04:43:35.03995756 +0000 UTC m=+1043602.659612078: ☑️ agreed by lance6716.

Rustin170506

Looks good to me. Thanks!

Rustin170506 · 2024-12-02T05:09:26Z

pkg/planner/core/memtable_infoschema_extractor.go

 	if len(tableNames) > 0 {
 		tableNames = filterSchemaObjectByRegexp(e, ec.table, tableNames, extractStrCIStr)
+		schemas := e.ListSchemas(is)
 		return findTableAndSchemaByName(ctx, is, schemas, tableNames)
 	}
+	schemas := e.ListSchemas(is)
 	return listTablesForEachSchema(ctx, e, is, schemas)


Suggested change

if len(tableNames) > 0 {

tableNames = filterSchemaObjectByRegexp(e, ec.table, tableNames, extractStrCIStr)

schemas := e.ListSchemas(is)

return findTableAndSchemaByName(ctx, is, schemas, tableNames)

}

schemas := e.ListSchemas(is)

return listTablesForEachSchema(ctx, e, is, schemas)

schemas := e.ListSchemas(is)

if len(tableNames) > 0 {

tableNames = filterSchemaObjectByRegexp(e, ec.table, tableNames, extractStrCIStr)

return findTableAndSchemaByName(ctx, is, schemas, tableNames)

}

return listTablesForEachSchema(ctx, e, is, schemas)

ti-chi-bot · 2024-12-02T05:10:09Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: joechenrh, lance6716, Rustin170506, tiancaiamao

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/planner/OWNERS~~ [Rustin170506]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2024-12-02T05:13:30Z

In response to a cherrypick label: new pull request created to branch release-8.5: #57870.

close #57779, close #57783

planner: use TableInfo.DBID to locate schema

325047b

tangenta added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 28, 2024

ti-chi-bot bot added do-not-merge/needs-triage-completed do-not-merge/needs-tests-checked release-note-none Denotes a PR that doesn't merit a release note. sig/planner SIG: Planner size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 28, 2024

ti-chi-bot bot removed the do-not-merge/needs-triage-completed label Nov 28, 2024

fix test

50e83b1

fix TestSameTableNameInTwoSchemas

f5a4818

ti-chi-bot bot removed the do-not-merge/needs-tests-checked label Nov 29, 2024

joechenrh reviewed Nov 29, 2024

View reviewed changes

tangenta commented Nov 29, 2024

View reviewed changes

tangenta removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2024

use SchemaNameByTableID

d63917c

joechenrh reviewed Nov 29, 2024

View reviewed changes

pkg/planner/core/memtable_infoschema_extractor.go Outdated Show resolved Hide resolved

joechenrh reviewed Nov 29, 2024

View reviewed changes

pkg/planner/core/memtable_infoschema_extractor.go Outdated Show resolved Hide resolved

tangenta added 3 commits November 29, 2024 17:29

address comment

1b95a99

Merge remote-tracking branch 'upstream/master' into fix-57783

19e223f

fix TestDelayMetrics

48ea22f

joechenrh approved these changes Dec 2, 2024

View reviewed changes

tangenta added the needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. label Dec 2, 2024

tiancaiamao approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 2, 2024

address comment

7dedf20

lance6716 approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Dec 2, 2024

Rustin170506 approved these changes Dec 2, 2024

View reviewed changes

ti-chi-bot bot added the approved label Dec 2, 2024

ti-chi-bot bot merged commit 6d74071 into pingcap:master Dec 2, 2024
24 checks passed

ti-chi-bot mentioned this pull request Dec 2, 2024

planner: use TableInfo.DBID to locate schema (#57785) #57870

Merged

13 tasks

ti-chi-bot bot pushed a commit that referenced this pull request Dec 2, 2024

planner: use TableInfo.DBID to locate schema (#57785) (#57870)

a248b96

close #57779, close #57783

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

planner: use TableInfo.DBID to locate schema #57785

planner: use TableInfo.DBID to locate schema #57785

tangenta commented Nov 28, 2024 •

edited

Loading

tiprow bot commented Nov 28, 2024

codecov bot commented Nov 28, 2024 •

edited

Loading

joechenrh Nov 29, 2024 •

edited

Loading

tangenta Nov 29, 2024

joechenrh Nov 29, 2024 •

edited

Loading

tangenta Nov 29, 2024

joechenrh Nov 29, 2024

ti-chi-bot bot commented Dec 2, 2024

tiancaiamao Dec 2, 2024

ti-chi-bot bot commented Dec 2, 2024

Rustin170506 left a comment

Rustin170506 Dec 2, 2024

ti-chi-bot bot commented Dec 2, 2024

ti-chi-bot commented Dec 2, 2024

planner: use TableInfo.DBID to locate schema #57785

planner: use TableInfo.DBID to locate schema #57785

Conversation

tangenta commented Nov 28, 2024 • edited Loading

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented Nov 28, 2024

codecov bot commented Nov 28, 2024 • edited Loading

Codecov Report

joechenrh Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

tangenta Nov 29, 2024

Choose a reason for hiding this comment

joechenrh Nov 29, 2024 • edited Loading

Choose a reason for hiding this comment

tangenta Nov 29, 2024

Choose a reason for hiding this comment

joechenrh Nov 29, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 2, 2024

tiancaiamao Dec 2, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 2, 2024

[LGTM Timeline notifier]

Rustin170506 left a comment

Choose a reason for hiding this comment

Rustin170506 Dec 2, 2024

Choose a reason for hiding this comment

ti-chi-bot bot commented Dec 2, 2024

ti-chi-bot commented Dec 2, 2024

tangenta commented Nov 28, 2024 •

edited

Loading

codecov bot commented Nov 28, 2024 •

edited

Loading

joechenrh Nov 29, 2024 •

edited

Loading

joechenrh Nov 29, 2024 •

edited

Loading