Skip to content

feat(optimizer): optimizer trace log #17746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Apr 10, 2025
Merged

feat(optimizer): optimizer trace log #17746

merged 7 commits into from
Apr 10, 2025

Conversation

BohuTANG
Copy link
Member

@BohuTANG BohuTANG commented Apr 10, 2025 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Add enable_optimizer_trace setting, default is disable
set enable_optimizer_trace =1;
-- TPC-H 10
select c_custkey, c_name, sum(l_extendedprice * (1 - l_discount)) as revenue, c_acctbal, n_name, c_address, c_phone, c_comment from customer, orders, lineitem, nation where c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate >= to_date('1993-10-01') and o_orderdate < add_months(to_date('1993-10-01'), 3) and l_returnflag = 'R' and c_nationkey = n_nationkey group by c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment order by revenue desc limit 20;

Optimizer trace logs in the log file:

[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ Optimizer Execution Summary ================

Applied Optimizers:
  - [4] PullUpFilterOptimizer (execution time: 104.41µs)
  - [5] RecursiveOptimizer[EliminateSort,EliminateUnion,...(28)] (execution time: 1.13ms)
  - [6] RecursiveOptimizer[SplitAggregate] (execution time: 25.06µs)
  - [7] DPhpy (execution time: 499.10µs)
  - [12] RecursiveOptimizer[EliminateEvalScalar] (execution time: 65.74µs)

Non-Applied Optimizers:
  - [0] SubqueryRewriter (execution time: 4.48µs)
  - [1] RuleStatsAggregateOptimizer (execution time: 11.59µs)
  - [2] CollectStatisticsOptimizer (execution time: 595.34ms)
  - [3] RuleNormalizeAggregateOptimizer (execution time: 11.29µs)
  - [8] SingleToInnerOptimizer (execution time: 4.99µs)
  - [9] DeduplicateJoinConditionOptimizer (execution time: 14.41µs)
  - [10] RecursiveOptimizer[CommuteJoin] (execution time: 24.30µs)
  - [11] CascadesOptimizer (execution time: 336.28µs)

Summary: 5/13 optimizers had effect (total execution time: 597.57ms)

[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ Applied Optimizer [12/13]: RecursiveOptimizer[EliminateEvalScalar] (execution time: 65.74µs) ================
 Limit
 ├── limit: [20]
 ├── offset: [0]
 └── Sort
     ├── sort keys: [derived.sum(l_extendedprice * (1 - l_discount)) (#38) DESC NULLS LAST]
     ├── limit: [20]
-    └── EvalScalar
-        ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), sum(l_extendedprice * (1 - l_discount)) (#38) AS (#38)]
-        └── Aggregate(Final)
+    └── Aggregate(Final)
+        ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
+        ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
+        └── Aggregate(Partial)
             ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
             ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
-            └── Aggregate(Partial)
-                ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
-                ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
-                └── EvalScalar
-                    ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
+            └── EvalScalar
+                ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
+                └── Join(Inner)
+                    ├── build keys: [orders.o_orderkey (#8)]
+                    ├── probe keys: [lineitem.l_orderkey (#17)]
+                    ├── other filters: []
+                    ├── Filter
+                    │   ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                    │   └── Scan
+                    │       ├── table: tpch_sf100.lineitem (#2)
+                    │       ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                    │       ├── order by: []
+                    │       └── limit: NONE
                     └── Join(Inner)
-                        ├── build keys: [orders.o_orderkey (#8)]
-                        ├── probe keys: [lineitem.l_orderkey (#17)]
+                        ├── build keys: [nation.n_nationkey (#33)]
+                        ├── probe keys: [customer.c_nationkey (#3)]
                         ├── other filters: []
-                        ├── Filter
-                        │   ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
-                        │   └── Scan
-                        │       ├── table: tpch_sf100.lineitem (#2)
-                        │       ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
-                        │       ├── order by: []
-                        │       └── limit: NONE
-                        └── Join(Inner)
-                            ├── build keys: [nation.n_nationkey (#33)]
-                            ├── probe keys: [customer.c_nationkey (#3)]
-                            ├── other filters: []
-                            ├── Join(Inner)
-                            │   ├── build keys: [orders.o_custkey (#9)]
-                            │   ├── probe keys: [customer.c_custkey (#0)]
-                            │   ├── other filters: []
-                            │   ├── Scan
-                            │   │   ├── table: tpch_sf100.customer (#0)
-                            │   │   ├── filters: []
-                            │   │   ├── order by: []
-                            │   │   └── limit: NONE
-                            │   └── Filter
-                            │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
-                            │       └── Scan
-                            │           ├── table: tpch_sf100.orders (#1)
-                            │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
-                            │           ├── order by: []
-                            │           └── limit: NONE
-                            └── Scan
-                                ├── table: tpch_sf100.nation (#3)
-                                ├── filters: []
-                                ├── order by: []
-                                └── limit: NONE
+                        ├── Join(Inner)
+                        │   ├── build keys: [orders.o_custkey (#9)]
+                        │   ├── probe keys: [customer.c_custkey (#0)]
+                        │   ├── other filters: []
+                        │   ├── Scan
+                        │   │   ├── table: tpch_sf100.customer (#0)
+                        │   │   ├── filters: []
+                        │   │   ├── order by: []
+                        │   │   └── limit: NONE
+                        │   └── Filter
+                        │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                        │       └── Scan
+                        │           ├── table: tpch_sf100.orders (#1)
+                        │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                        │           ├── order by: []
+                        │           └── limit: NONE
+                        └── Scan
+                            ├── table: tpch_sf100.nation (#3)
+                            ├── filters: []
+                            ├── order by: []
+                            └── limit: NONE


[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [11/13]: CascadesOptimizer (execution time: 336.28µs) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [10/13]: RecursiveOptimizer[CommuteJoin] (execution time: 24.30µs) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [9/13]: DeduplicateJoinConditionOptimizer (execution time: 14.41µs) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [8/13]: SingleToInnerOptimizer (execution time: 4.99µs) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ Applied Optimizer [7/13]: DPhpy (execution time: 499.10µs) ================
 Limit
 ├── limit: [20]
 ├── offset: [0]
 └── Sort
     ├── sort keys: [derived.sum(l_extendedprice * (1 - l_discount)) (#38) DESC NULLS LAST]
     ├── limit: [20]
     └── EvalScalar
         ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), sum(l_extendedprice * (1 - l_discount)) (#38) AS (#38)]
         └── Aggregate(Final)
             ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
             ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
             └── Aggregate(Partial)
                 ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
                 ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
                 └── EvalScalar
                     ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
                     └── Join(Inner)
-                        ├── build keys: [nation.n_nationkey (#33)]
-                        ├── probe keys: [customer.c_nationkey (#3)]
+                        ├── build keys: [orders.o_orderkey (#8)]
+                        ├── probe keys: [lineitem.l_orderkey (#17)]
                         ├── other filters: []
-                        ├── Join(Inner)
-                        │   ├── build keys: [lineitem.l_orderkey (#17)]
-                        │   ├── probe keys: [orders.o_orderkey (#8)]
-                        │   ├── other filters: []
-                        │   ├── Join(Inner)
-                        │   │   ├── build keys: [orders.o_custkey (#9)]
-                        │   │   ├── probe keys: [customer.c_custkey (#0)]
-                        │   │   ├── other filters: []
-                        │   │   ├── Scan
-                        │   │   │   ├── table: tpch_sf100.customer (#0)
-                        │   │   │   ├── filters: []
-                        │   │   │   ├── order by: []
-                        │   │   │   └── limit: NONE
-                        │   │   └── Filter
-                        │   │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
-                        │   │       └── Scan
-                        │   │           ├── table: tpch_sf100.orders (#1)
-                        │   │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
-                        │   │           ├── order by: []
-                        │   │           └── limit: NONE
-                        │   └── Filter
+                        ├── Filter
+                        │   ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                        │   └── Scan
+                        │       ├── table: tpch_sf100.lineitem (#2)
                         │       ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
-                        │       └── Scan
-                        │           ├── table: tpch_sf100.lineitem (#2)
-                        │           ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
-                        │           ├── order by: []
-                        │           └── limit: NONE
-                        └── Scan
-                            ├── table: tpch_sf100.nation (#3)
-                            ├── filters: []
-                            ├── order by: []
-                            └── limit: NONE
+                        │       ├── order by: []
+                        │       └── limit: NONE
+                        └── Join(Inner)
+                            ├── build keys: [nation.n_nationkey (#33)]
+                            ├── probe keys: [customer.c_nationkey (#3)]
+                            ├── other filters: []
+                            ├── Join(Inner)
+                            │   ├── build keys: [orders.o_custkey (#9)]
+                            │   ├── probe keys: [customer.c_custkey (#0)]
+                            │   ├── other filters: []
+                            │   ├── Scan
+                            │   │   ├── table: tpch_sf100.customer (#0)
+                            │   │   ├── filters: []
+                            │   │   ├── order by: []
+                            │   │   └── limit: NONE
+                            │   └── Filter
+                            │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                            │       └── Scan
+                            │           ├── table: tpch_sf100.orders (#1)
+                            │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                            │           ├── order by: []
+                            │           └── limit: NONE
+                            └── Scan
+                                ├── table: tpch_sf100.nation (#3)
+                                ├── filters: []
+                                ├── order by: []
+                                └── limit: NONE

[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ Applied Optimizer [6/13]: RecursiveOptimizer[SplitAggregate] (execution time: 25.06µs) ================
 Limit
 ├── limit: [20]
 ├── offset: [0]
 └── Sort
     ├── sort keys: [derived.sum(l_extendedprice * (1 - l_discount)) (#38) DESC NULLS LAST]
     ├── limit: [20]
     └── EvalScalar
         ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), sum(l_extendedprice * (1 - l_discount)) (#38) AS (#38)]
-        └── Aggregate(Initial)
+        └── Aggregate(Final)
             ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
             ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
-            └── EvalScalar
-                ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
-                └── Join(Inner)
-                    ├── build keys: [nation.n_nationkey (#33)]
-                    ├── probe keys: [customer.c_nationkey (#3)]
-                    ├── other filters: []
-                    ├── Join(Inner)
-                    │   ├── build keys: [lineitem.l_orderkey (#17)]
-                    │   ├── probe keys: [orders.o_orderkey (#8)]
-                    │   ├── other filters: []
-                    │   ├── Join(Inner)
-                    │   │   ├── build keys: [orders.o_custkey (#9)]
-                    │   │   ├── probe keys: [customer.c_custkey (#0)]
-                    │   │   ├── other filters: []
-                    │   │   ├── Scan
-                    │   │   │   ├── table: tpch_sf100.customer (#0)
-                    │   │   │   ├── filters: []
-                    │   │   │   ├── order by: []
-                    │   │   │   └── limit: NONE
-                    │   │   └── Filter
-                    │   │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
-                    │   │       └── Scan
-                    │   │           ├── table: tpch_sf100.orders (#1)
-                    │   │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
-                    │   │           ├── order by: []
-                    │   │           └── limit: NONE
-                    │   └── Filter
-                    │       ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
-                    │       └── Scan
-                    │           ├── table: tpch_sf100.lineitem (#2)
-                    │           ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
-                    │           ├── order by: []
-                    │           └── limit: NONE
-                    └── Scan
-                        ├── table: tpch_sf100.nation (#3)
-                        ├── filters: []
-                        ├── order by: []
-                        └── limit: NONE
+            └── Aggregate(Partial)
+                ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
+                ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
+                └── EvalScalar
+                    ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
+                    └── Join(Inner)
+                        ├── build keys: [nation.n_nationkey (#33)]
+                        ├── probe keys: [customer.c_nationkey (#3)]
+                        ├── other filters: []
+                        ├── Join(Inner)
+                        │   ├── build keys: [lineitem.l_orderkey (#17)]
+                        │   ├── probe keys: [orders.o_orderkey (#8)]
+                        │   ├── other filters: []
+                        │   ├── Join(Inner)
+                        │   │   ├── build keys: [orders.o_custkey (#9)]
+                        │   │   ├── probe keys: [customer.c_custkey (#0)]
+                        │   │   ├── other filters: []
+                        │   │   ├── Scan
+                        │   │   │   ├── table: tpch_sf100.customer (#0)
+                        │   │   │   ├── filters: []
+                        │   │   │   ├── order by: []
+                        │   │   │   └── limit: NONE
+                        │   │   └── Filter
+                        │   │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                        │   │       └── Scan
+                        │   │           ├── table: tpch_sf100.orders (#1)
+                        │   │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                        │   │           ├── order by: []
+                        │   │           └── limit: NONE
+                        │   └── Filter
+                        │       ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                        │       └── Scan
+                        │           ├── table: tpch_sf100.lineitem (#2)
+                        │           ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                        │           ├── order by: []
+                        │           └── limit: NONE
+                        └── Scan
+                            ├── table: tpch_sf100.nation (#3)
+                            ├── filters: []
+                            ├── order by: []
+                            └── limit: NONE

[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ Applied Optimizer [5/13]: RecursiveOptimizer[EliminateSort,EliminateUnion,...(28)] (execution time: 1.13ms) ================
 Limit
 ├── limit: [20]
 ├── offset: [0]
 └── Sort
     ├── sort keys: [derived.sum(l_extendedprice * (1 - l_discount)) (#38) DESC NULLS LAST]
-    ├── limit: [NONE]
+    ├── limit: [20]
     └── EvalScalar
         ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), sum(l_extendedprice * (1 - l_discount)) (#38) AS (#38)]
         └── Aggregate(Initial)
             ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
             ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
-            └── Filter
-                ├── filters: [eq(customer.c_custkey (#0), orders.o_custkey (#39)), eq(lineitem.l_orderkey (#40), orders.o_orderkey (#41)), eq(customer.c_nationkey (#44), nation.n_nationkey (#45)), gte(orders.o_orderdate (#42), '1993-10-01'), lt(orders.o_orderdate (#42), '1994-01-01'), eq(lineitem.l_returnflag (#43), 'R')]
-                └── EvalScalar
-                    ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
-                    └── Join(Cross)
-                        ├── build keys: []
-                        ├── probe keys: []
-                        ├── other filters: []
-                        ├── Join(Cross)
-                        │   ├── build keys: []
-                        │   ├── probe keys: []
-                        │   ├── other filters: []
-                        │   ├── Join(Cross)
-                        │   │   ├── build keys: []
-                        │   │   ├── probe keys: []
-                        │   │   ├── other filters: []
-                        │   │   ├── Scan
-                        │   │   │   ├── table: tpch_sf100.customer (#0)
-                        │   │   │   ├── filters: []
-                        │   │   │   ├── order by: []
-                        │   │   │   └── limit: NONE
-                        │   │   └── Scan
-                        │   │       ├── table: tpch_sf100.orders (#1)
-                        │   │       ├── filters: []
-                        │   │       ├── order by: []
-                        │   │       └── limit: NONE
-                        │   └── Scan
-                        │       ├── table: tpch_sf100.lineitem (#2)
-                        │       ├── filters: []
-                        │       ├── order by: []
-                        │       └── limit: NONE
-                        └── Scan
-                            ├── table: tpch_sf100.nation (#3)
-                            ├── filters: []
-                            ├── order by: []
-                            └── limit: NONE
+            └── EvalScalar
+                ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
+                └── Join(Inner)
+                    ├── build keys: [nation.n_nationkey (#33)]
+                    ├── probe keys: [customer.c_nationkey (#3)]
+                    ├── other filters: []
+                    ├── Join(Inner)
+                    │   ├── build keys: [lineitem.l_orderkey (#17)]
+                    │   ├── probe keys: [orders.o_orderkey (#8)]
+                    │   ├── other filters: []
+                    │   ├── Join(Inner)
+                    │   │   ├── build keys: [orders.o_custkey (#9)]
+                    │   │   ├── probe keys: [customer.c_custkey (#0)]
+                    │   │   ├── other filters: []
+                    │   │   ├── Scan
+                    │   │   │   ├── table: tpch_sf100.customer (#0)
+                    │   │   │   ├── filters: []
+                    │   │   │   ├── order by: []
+                    │   │   │   └── limit: NONE
+                    │   │   └── Filter
+                    │   │       ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                    │   │       └── Scan
+                    │   │           ├── table: tpch_sf100.orders (#1)
+                    │   │           ├── filters: [gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01')]
+                    │   │           ├── order by: []
+                    │   │           └── limit: NONE
+                    │   └── Filter
+                    │       ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                    │       └── Scan
+                    │           ├── table: tpch_sf100.lineitem (#2)
+                    │           ├── filters: [eq(lineitem.l_returnflag (#25), 'R')]
+                    │           ├── order by: []
+                    │           └── limit: NONE
+                    └── Scan
+                        ├── table: tpch_sf100.nation (#3)
+                        ├── filters: []
+                        ├── order by: []
+                        └── limit: NONE

[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ Applied Optimizer [4/13]: PullUpFilterOptimizer (execution time: 104.41µs) ================
 Limit
 ├── limit: [20]
 ├── offset: [0]
 └── Sort
     ├── sort keys: [derived.sum(l_extendedprice * (1 - l_discount)) (#38) DESC NULLS LAST]
     ├── limit: [NONE]
     └── EvalScalar
         ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), sum(l_extendedprice * (1 - l_discount)) (#38) AS (#38)]
         └── Aggregate(Initial)
             ├── group items: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_acctbal (#5) AS (#5), customer.c_phone (#4) AS (#4), nation.n_name (#34) AS (#34), customer.c_address (#2) AS (#2), customer.c_comment (#7) AS (#7)]
             ├── aggregate functions: [sum(l_extendedprice * (1 - l_discount)) AS (#38)]
-            └── EvalScalar
-                ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37)]
-                └── Filter
-                    ├── filters: [eq(customer.c_custkey (#0), orders.o_custkey (#9)), eq(lineitem.l_orderkey (#17), orders.o_orderkey (#8)), gte(orders.o_orderdate (#12), '1993-10-01'), lt(orders.o_orderdate (#12), '1994-01-01'), eq(lineitem.l_returnflag (#25), 'R'), eq(customer.c_nationkey (#3), nation.n_nationkey (#33))]
+            └── Filter
+                ├── filters: [eq(customer.c_custkey (#0), orders.o_custkey (#39)), eq(lineitem.l_orderkey (#40), orders.o_orderkey (#41)), eq(customer.c_nationkey (#44), nation.n_nationkey (#45)), gte(orders.o_orderdate (#42), '1993-10-01'), lt(orders.o_orderdate (#42), '1994-01-01'), eq(lineitem.l_returnflag (#43), 'R')]
+                └── EvalScalar
+                    ├── scalars: [customer.c_custkey (#0) AS (#0), customer.c_name (#1) AS (#1), customer.c_address (#2) AS (#2), customer.c_phone (#4) AS (#4), customer.c_acctbal (#5) AS (#5), customer.c_comment (#7) AS (#7), nation.n_name (#34) AS (#34), multiply(lineitem.l_extendedprice (#22), minus(1, lineitem.l_discount (#23))) AS (#37), orders.o_custkey (#9) AS (#39), lineitem.l_orderkey (#17) AS (#40), orders.o_orderkey (#8) AS (#41), orders.o_orderdate (#12) AS (#42), lineitem.l_returnflag (#25) AS (#43), customer.c_nationkey (#3) AS (#44), nation.n_nationkey (#33) AS (#45)]
                     └── Join(Cross)
                         ├── build keys: []
                         ├── probe keys: []
                         ├── other filters: []
                         ├── Join(Cross)
                         │   ├── build keys: []
                         │   ├── probe keys: []
                         │   ├── other filters: []
                         │   ├── Join(Cross)
                         │   │   ├── build keys: []
                         │   │   ├── probe keys: []
                         │   │   ├── other filters: []
                         │   │   ├── Scan
                         │   │   │   ├── table: tpch_sf100.customer (#0)
                         │   │   │   ├── filters: []
                         │   │   │   ├── order by: []
                         │   │   │   └── limit: NONE
                         │   │   └── Scan
                         │   │       ├── table: tpch_sf100.orders (#1)
                         │   │       ├── filters: []
                         │   │       ├── order by: []
                         │   │       └── limit: NONE
                         │   └── Scan
                         │       ├── table: tpch_sf100.lineitem (#2)
                         │       ├── filters: []
                         │       ├── order by: []
                         │       └── limit: NONE
                         └── Scan
                             ├── table: tpch_sf100.nation (#3)
                             ├── filters: []
                             ├── order by: []
                             └── limit: NONE

[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [3/13]: RuleNormalizeAggregateOptimizer (execution time: 11.29µs) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [2/13]: CollectStatisticsOptimizer (execution time: 595.34ms) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [1/13]: RuleStatsAggregateOptimizer (execution time: 11.59µs) ================
[0] INFO - 6aaa3330-253d-4c17-bbaf-a63aad3a76f1 -  ================ No Effect Optimizer [0/13]: SubqueryRewriter (execution time: 4.48µs) ================

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Apr 10, 2025
@BohuTANG BohuTANG changed the title feat(optimizer): optimizer trace feat(optimizer): optimizer trace log Apr 10, 2025
@BohuTANG BohuTANG added the ci-cloud Build docker image for cloud test label Apr 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17746-39aa97e-1744257812

note: this image tag is only available for internal use.

@BohuTANG BohuTANG marked this pull request as ready for review April 10, 2025 04:22
@BohuTANG BohuTANG requested a review from zhang2014 April 10, 2025 04:22
@BohuTANG BohuTANG added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Apr 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17746-1998cd0-1744264765

note: this image tag is only available for internal use.

@BohuTANG BohuTANG force-pushed the dev-optimizer-trace branch from 7066f6f to ac1075c Compare April 10, 2025 06:20
@BohuTANG BohuTANG added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Apr 10, 2025
@github-actions GitHub Actions
Copy link
Contributor

Docker Image for PR

  • tag: pr-17746-614d478-1744267827

note: this image tag is only available for internal use.

@BohuTANG BohuTANG merged commit 9b6639e into databendlabs:main Apr 10, 2025
90 checks passed
@BohuTANG BohuTANG mentioned this pull request Apr 10, 2025
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhang2014 zhang2014 zhang2014 approved these changes

Assignees
No one assigned
Labels
ci-cloud Build docker image for cloud test pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BohuTANG @zhang2014