[opt](nereids) support pushing down aggr distinct through join #43380

xzj7019 · 2024-11-06T13:00:27Z

What problem does this PR solve?

support pushing down agg distinct through join on one side.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No colde files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
Release note

None

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-06T13:00:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xzj7019 · 2024-11-28T13:37:30Z

run buildall

xzj7019 · 2024-11-29T02:41:45Z

run buildall

feiniaofeiafei · 2024-11-29T04:19:33Z

...n/java/org/apache/doris/nereids/rules/rewrite/PushDownAggWithDistinctThroughJoinOneSide.java

+                        .when(agg -> agg.child().child().getOtherJoinConjuncts().isEmpty())
+                        .when(agg -> !agg.isGenerated())
+                        .whenNot(agg -> agg.getAggregateFunctions().isEmpty())
+                        .whenNot(agg -> agg.child().children().stream().anyMatch(p -> p instanceof LogicalAggregate))


agg -> agg.child().children().stream().anyMatch(p -> p instanceof LogicalAggregate) , this is always false.

thx, should be changed to .whenNot(agg -> agg.child()
.child(0).children().stream().anyMatch(p -> p instanceof LogicalAggregate))

feiniaofeiafei · 2024-11-29T04:20:39Z

...n/java/org/apache/doris/nereids/rules/rewrite/PushDownAggWithDistinctThroughJoinOneSide.java

+            leftFuncs.forEach(func -> {
+                Alias alias = func.alias("PDADT_" + func.getName());
+                leftSlotToOutput.put((Slot) func.child(0), alias);
+            });


map leftSlotToOutput value seems to be useless

xzj7019 · 2024-11-29T06:51:24Z

run buildall

github-actions · 2024-11-29T06:54:20Z

PR approved by anyone and no changes requested.

xzj7019 · 2024-11-29T08:16:44Z

run buildall

xzj7019 · 2024-11-29T11:35:50Z

run cloud_p0

xzj7019 · 2024-12-02T01:55:42Z

run cloud_p0

englefly · 2024-12-02T02:08:16Z

...n/java/org/apache/doris/nereids/rules/rewrite/PushDownAggWithDistinctThroughJoinOneSide.java

+    public List<Rule> buildRules() {
+        return ImmutableList.of(
+                logicalAggregate(logicalProject(innerLogicalJoin()))
+                        .when(agg -> agg.child().isAllSlots())


agg.child().isAllSlots()
is this restriction too strong?

aggr with distinct pushing down has special rewrite logic and restrict this case carefully by safety

englefly · 2024-12-02T02:09:10Z

...n/java/org/apache/doris/nereids/rules/rewrite/PushDownAggWithDistinctThroughJoinOneSide.java

+        return ImmutableList.of(
+                logicalAggregate(logicalProject(innerLogicalJoin()))
+                        .when(agg -> agg.child().isAllSlots())
+                        .when(agg -> agg.child().child().getOtherJoinConjuncts().isEmpty())


"getOtherJoinConjuncts().isEmpty()"
how about remove this?

leave this clean up task for the whole series of aggr pushing down rules

englefly · 2024-12-02T02:18:51Z

why it is called oneSide? it could push down through both sides

xzj7019 · 2024-12-02T03:13:48Z

run cloud_p0

xzj7019 · 2024-12-02T03:29:37Z

run buildall

xzj7019 · 2024-12-02T09:22:39Z

run cloud_p0

xzj7019 · 2024-12-02T09:29:07Z

run buildall

xzj7019 · 2024-12-04T07:15:23Z

run buildall

xzj7019 · 2024-12-04T10:59:54Z

run p0

xzj7019 · 2024-12-04T11:25:30Z

run p0

github-actions · 2024-12-05T01:57:01Z

PR approved by at least one committer and no changes requested.

… join #43380 (#45067) Cherry-picked from #43380 Co-authored-by: xzj7019 <xiongzhongjian@selectdb.com>

xzj7019 marked this pull request as draft November 6, 2024 13:00

xzj7019 marked this pull request as ready for review November 19, 2024 06:16

xzj7019 marked this pull request as draft November 19, 2024 06:16

xzj7019 force-pushed the pushdown_agg_distinct branch 2 times, most recently from 7a772d1 to 29c1de5 Compare November 28, 2024 13:37

xzj7019 marked this pull request as ready for review November 28, 2024 13:37

xzj7019 changed the title ~~[opt](nereids) support pushdown agg distinct through join~~ [opt](nereids) support pushing down aggr distinct through join Nov 28, 2024

xzj7019 force-pushed the pushdown_agg_distinct branch from 29c1de5 to 964514c Compare November 29, 2024 02:40

feiniaofeiafei reviewed Nov 29, 2024

View reviewed changes

xzj7019 force-pushed the pushdown_agg_distinct branch from 964514c to e99aa8c Compare November 29, 2024 06:50

feiniaofeiafei approved these changes Nov 29, 2024

View reviewed changes

github-actions bot added the reviewed label Nov 29, 2024

xzj7019 force-pushed the pushdown_agg_distinct branch from e99aa8c to 8076eb0 Compare November 29, 2024 08:16

englefly reviewed Dec 2, 2024

View reviewed changes

xzj7019 force-pushed the pushdown_agg_distinct branch from 8076eb0 to b243038 Compare December 2, 2024 03:28

xzj7019 force-pushed the pushdown_agg_distinct branch from bf3e09d to 5e048db Compare December 2, 2024 09:27

xzj7019 added 15 commits December 4, 2024 15:14

[opt](nereids) support pushdown agg distinct through join

fd02225

[opt](nereids) support pushdown agg distinct through join

d8aae1d

[opt](nereids) support pushdown agg distinct through join

0ed5e3f

[opt](nereids) support pushdown agg distinct through join

3fed78c

[opt](nereids) support pushdown agg distinct through join

2b9e6ad

[opt](nereids) support pushdown agg distinct through join

eafab7b

[opt](nereids) support pushdown agg distinct through join

f961f14

[opt](nereids) support pushdown agg distinct through join

02ba75d

[opt](nereids) support pushdown agg distinct through join

805c312

[opt](nereids) support pushdown agg distinct through join

67049f7

[opt](nereids) support pushdown agg distinct through join

2530ff0

[opt](nereids) support pushdown agg distinct through join

6d4e705

[opt](nereids) support pushdown agg distinct through join

2c4c431

[opt](nereids) support pushdown agg distinct through join

5687192

[opt](nereids) support pushdown agg distinct through join

9e4d53c

xzj7019 force-pushed the pushdown_agg_distinct branch from c85ccfa to 9e4d53c Compare December 4, 2024 07:15

englefly approved these changes Dec 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2024

morrySnow merged commit bf4ba66 into apache:master Dec 5, 2024
14 checks passed

morrySnow added the dev/3.0.x label Dec 5, 2024

github-actions bot pushed a commit that referenced this pull request Dec 5, 2024

[opt](nereids) support pushing down aggr distinct through join (#43380)

986af64

github-actions bot mentioned this pull request Dec 5, 2024

branch-3.0: [opt](nereids) support pushing down aggr distinct through join #43380 #45067

Merged

morrySnow pushed a commit that referenced this pull request Dec 10, 2024

branch-3.0: [opt](nereids) support pushing down aggr distinct through…

243561d

… join #43380 (#45067) Cherry-picked from #43380 Co-authored-by: xzj7019 <xiongzhongjian@selectdb.com>

morrySnow added dev/3.0.4-merged and removed dev/3.0.x labels Dec 10, 2024

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

[opt](nereids) support pushing down aggr distinct through join #43380

[opt](nereids) support pushing down aggr distinct through join #43380

Uh oh!

Conversation

xzj7019 commented Nov 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

xzj7019 commented Nov 28, 2024

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

github-actions bot commented Nov 29, 2024

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

englefly commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 4, 2024

Uh oh!

xzj7019 commented Dec 4, 2024

Uh oh!

xzj7019 commented Dec 4, 2024

Uh oh!

github-actions bot commented Dec 5, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xzj7019 commented Nov 6, 2024 •

edited

Loading