Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deep nested OR expressions caused compile oom #42788

Closed
chrysan opened this issue Apr 3, 2023 · 5 comments · Fixed by #46450
Closed

Deep nested OR expressions caused compile oom #42788

chrysan opened this issue Apr 3, 2023 · 5 comments · Fixed by #46450
Assignees
@chrysan
Labels
affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 severity/major sig/planner SIG: Planner type/bug The issue is confirmed as a bug.

Comments

@chrysan
Copy link
Contributor

chrysan commented Apr 3, 2023 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Run a sql with deep nested OR expressions.
img_v2_68600fb7-25ba-4b4f-aa55-d9252b69f4fg

We need:

  1. memory track and threshold

2. What did you expect to see? (Required)

SQL runs normally or is killed due to memory quota.

3. What did you see instead (Required)

TiDB oom.
If we look into the oom dump:
mqqzwGZ14c
y8cdwVNoY5

4. What is your TiDB version? (Required)

v6.5
@chrysan chrysan added the type/bug The issue is confirmed as a bug. label Apr 3, 2023
@chrysan
Copy link
Contributor Author

chrysan commented Apr 3, 2023

We need:

  1. track the memory of the exprs' hashCode and make the sql killed when exceeds memory threshold
  2. need a new implementation of ReHashCode which is more memory efficient

@chrysan chrysan self-assigned this Apr 3, 2023
@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 labels Apr 3, 2023
@tiancaiamao tiancaiamao mentioned this issue May 16, 2023
Open
@mayjiang0203
Copy link

mayjiang0203 commented Jun 16, 2023 •
edited
Loading

/label affects-6.5
from TCOC-994

@AilinKid AilinKid mentioned this issue Aug 28, 2023
Merged
12 tasks
@AilinKid
Copy link
Contributor

Before Fix

➜  core git:(fix-42788) ✗ go tool pprof m5.file
File: ___5BenchmarkSubstituteExpression_in_github_com_pingcap_tidb_planner_core.test
Build ID: ea7c603fe0cf5e18deac5bf65d36f115467fce80
Type: alloc_space
Time: Aug 28, 2023 at 3:58pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) list BenchmarkSubstituteExpression
Total: 1.40GB
ROUTINE ======================== github.com/pingcap/tidb/planner/core_test.BenchmarkSubstituteExpression in /home/arenatlx/go/src/github.com/pingcap/tidb/planner/core/rule_generate_column_substitute_test.go
         0   173.44MB (flat, cum) 12.12% of Total
         .          .     29:func BenchmarkSubstituteExpression(b *testing.B) {
         .   169.91MB     30:   store := testkit.CreateMockStore(b)
         .          .     31:   tk := testkit.NewTestKit(b, store)
         .          .     32:   tk.MustExec("use test")
         .          .     33:   tk.MustExec("drop table if exists tai")
         .   512.19kB     34:   tk.MustExec("create table tai(a varchar(256), b varchar(256), c int as (a+1), d int as (b+1))")
         .          .     35:   is := domain.GetDomain(tk.Session()).InfoSchema()
         .          .     36:   _, err := is.TableByName(model.NewCIStr("test"), model.NewCIStr("tai"))
         .          .     37:   require.NoError(b, err)
         .          .     38:   condition := "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     39:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     40:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     41:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     42:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     43:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     44:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     45:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     46:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     47:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     48:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     49:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     50:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     51:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     52:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     53:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     54:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     55:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     56:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     57:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     58:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     59:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     60:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     61:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     62:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     63:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     64:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     65:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     66:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     67:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     68:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     69:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     70:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     71:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     72:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     73:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     74:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     75:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     76:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     77:           "(tai.a='%s' AND tai.b='%s')"
         .          .     78:   addresses := make([]interface{}, 0, 90)
         .          .     79:   for i := 0; i < 80; i++ {
         .          .     80:           addresses = append(addresses, "0x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC510x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC510x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC510x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC51")
         .          .     81:   }
         .   520.04kB     82:   condition = fmt.Sprintf(condition, addresses...)
         .          .     83:   s := core.CreatePlannerSuite(tk.Session(), is)
         .          .     84:   ctx := context.Background()
         .          .     85:   sql := "select * from tai where " + condition
         .          .     86:   fmt.Println(sql)
         .          .     87:   stmt, err := s.GetParser().ParseOneStmt(sql, "", "")
         .          .     88:   require.NoError(b, err, sql)
         .   512.01kB     89:   p, _, err := core.BuildLogicalPlanForTest(ctx, s.GetCtx(), stmt, s.GetIS())
         .          .     90:   require.NoError(b, err)
         .          .     91:   selection := p.(core.LogicalPlan).Children()[0]
         .          .     92:   m := make(core.ExprColumnMap, len(selection.Schema().Columns))
         .          .     93:   for _, col := range selection.Schema().Columns {
         .          .     94:           if col.VirtualExpr != nil {
         .          .     95:                   m[col.VirtualExpr] = col
         .          .     96:           }
         .          .     97:   }
         .          .     98:   b.ResetTimer()
         .          .     99:   b.StartTimer()
         .          .    100:   for i := 0; i < b.N; i++ {
         .     2.02MB    101:           core.SubstituteExpression(selection.(*core.LogicalSelection).Conditions[0], selection, m, selection.Schema(), nil)
         .          .    102:   }
         .          .    103:   b.StopTimer()
         .          .    104:}

         
➜  core git:(fix-42788) ✗ go tool pprof m5.file
File: ___5BenchmarkSubstituteExpression_in_github_com_pingcap_tidb_planner_core.test
Build ID: dce9437cc5156c542bc642092b25b29de9b14d87
Type: alloc_space
Time: Aug 28, 2023 at 4:03pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) list BenchmarkSubstituteExpression
Total: 1.76GB
ROUTINE ======================== github.com/pingcap/tidb/planner/core_test.BenchmarkSubstituteExpression in /home/arenatlx/go/src/github.com/pingcap/tidb/planner/core/rule_generate_column_substitute_test.go
         0   220.74MB (flat, cum) 12.23% of Total
         .          .     29:func BenchmarkSubstituteExpression(b *testing.B) {
         .   220.74MB     30:   store := testkit.CreateMockStore(b)
         .          .     31:   tk := testkit.NewTestKit(b, store)
         .          .     32:   tk.MustExec("use test")
         .          .     33:   tk.MustExec("drop table if exists tai")
         .          .     34:   tk.MustExec("create table tai(a varchar(256), b varchar(256), c int as (a+1), d int as (b+1))")
         .          .     35:   is := domain.GetDomain(tk.Session()).InfoSchema()
(pprof) 
(pprof) 
(pprof) q

After Fix

➜  core git:(fix-42788) ✗ go tool pprof m5.file
File: ___5BenchmarkSubstituteExpression_in_github_com_pingcap_tidb_planner_core.test
Build ID: dce9437cc5156c542bc642092b25b29de9b14d87
Type: alloc_space
Time: Aug 28, 2023 at 4:04pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) list BenchmarkSubstituteExpression
Total: 1.41GB
ROUTINE ======================== github.com/pingcap/tidb/planner/core_test.BenchmarkSubstituteExpression in /home/arenatlx/go/src/github.com/pingcap/tidb/planner/core/rule_generate_column_substitute_test.go
         0   172.22MB (flat, cum) 11.90% of Total
         .          .     29:func BenchmarkSubstituteExpression(b *testing.B) {
         .   170.21MB     30:   store := testkit.CreateMockStore(b)
         .     1.01MB     31:   tk := testkit.NewTestKit(b, store)
         .          .     32:   tk.MustExec("use test")
         .          .     33:   tk.MustExec("drop table if exists tai")
         .          .     34:   tk.MustExec("create table tai(a varchar(256), b varchar(256), c int as (a+1), d int as (b+1))")
         .          .     35:   is := domain.GetDomain(tk.Session()).InfoSchema()
         .          .     36:   _, err := is.TableByName(model.NewCIStr("test"), model.NewCIStr("tai"))
         .          .     37:   require.NoError(b, err)
         .          .     38:   condition := "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     39:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     40:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     41:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     42:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     43:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     44:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     45:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     46:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     47:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     48:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     49:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     50:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     51:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     52:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     53:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     54:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     55:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     56:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     57:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     58:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     59:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     60:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     61:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     62:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     63:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     64:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     65:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     66:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     67:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     68:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     69:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     70:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     71:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     72:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     73:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     74:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     75:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     76:           "(tai.a='%s' AND tai.b='%s') OR" +
         .          .     77:           "(tai.a='%s' AND tai.b='%s')q"
         .          .     78:   addresses := make([]interface{}, 0, 90)
         .          .     79:   for i := 0; i < 80; i++ {
         .          .     80:           addresses = append(addresses, "0x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC510x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC510x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC510x6ab6Bf9117A8A9dd5a2FF203aa8a22457162fC51")
         .          .     81:   }
         .          .     82:   condition = fmt.Sprintf(condition, addresses...)
         .   520.04kB     83:   s := core.CreatePlannerSuite(tk.Session(), is)
         .          .     84:   ctx := context.Background()
         .          .     85:   sql := "select * from tai where " + condition
         .          .     86:   fmt.Println(sql)
         .          .     87:   stmt, err := s.GetParser().ParseOneStmt(sql, "", "")
         .          .     88:   require.NoError(b, err, sql)
         .   512.07kB     89:   p, _, err := core.BuildLogicalPlanForTest(ctx, s.GetCtx(), stmt, s.GetIS())
         .          .     90:   require.NoError(b, err)
         .          .     91:   selection := p.(core.LogicalPlan).Children()[0]
         .          .     92:   m := make(core.ExprColumnMap, len(selection.Schema().Columns))
         .          .     93:   for _, col := range selection.Schema().Columns {
         .          .     94:           if col.VirtualExpr != nil {
(pprof) 
(pprof) q

@XuHuaiyu
Copy link
Contributor

XuHuaiyu commented Sep 1, 2023

@AilinKid What's the difference between before and after

@AilinKid AilinKid added affects-5.4 This bug affects 5.4.x versions. affects-6.1 and removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.1 labels Sep 1, 2023
@AilinKid
Copy link
Contributor

AilinKid commented Sep 4, 2023 •
edited
Loading

@AilinKid What's the difference between before and after

for this case special, we could avoid hashcode computation of the entire expression tree from bottom to the root. the mem alloc of core.SubstituteExpression is no where to see

This was referenced Sep 4, 2023
Closed
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 25, 2023
@ti-chi-bot
planner: don't recompute the hashcode when generated column substitut…
e491060 
…ion doesn't happen (#46450) (#46630)

close #42788
ti-chi-bot bot pushed a commit that referenced this issue Oct 17, 2023
@ti-chi-bot
planner: don't recompute the hashcode when generated column substitut…
49d93b8 
…ion doesn't happen (#46450) (#46629)

close #42788
ti-chi-bot bot pushed a commit that referenced this issue Oct 30, 2023
@ti-chi-bot
planner: don't recompute the hashcode when generated column substitut…
f2faaa4 
…ion doesn't happen (#46450) (#46628)

close #42788
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chrysan chrysan

Labels
affects-5.4 This bug affects 5.4.x versions. affects-6.1 affects-6.5 affects-7.1 severity/major sig/planner SIG: Planner type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
6 participants
@chrysan @VelocityLight @XuHuaiyu @mayjiang0203 @AilinKid @ti-chi-bot