[SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number #31750
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #135787 has finished for PR 31750 at commit
|
Member
Author
|
cc @cloud-fan This is a part of #31691. To make that pr more clear. |
maropu
reviewed
Mar 6, 2021
| * This rule optimizes Limit operators by: | ||
| * 1. Eliminate [[Limit]] operators if it's child max row <= limit. | ||
| * 2. Combines two adjacent [[Limit]] operators into one, merging the | ||
| * 2. Eliminate [[GlobalLimit]] operators if it's child max row <= limit. |
Member
There was a problem hiding this comment.
nit: how about merging 1. and 2. like Eliminate [[Limit]]/[[GlobalLimit]] ...?
maropu
approved these changes
Mar 6, 2021
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
dongjoon-hyun
approved these changes
Mar 6, 2021
|
Test build #135830 has finished for PR 31750 at commit
|
Contributor
|
late LGTM |
wangyum
added a commit
that referenced
this pull request
May 26, 2023
… if its child max rows not larger than limit number (#1033) * [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number ### What changes were proposed in this pull request? This pr remove `GlobalLimit` operator if its child max rows not larger than limit number. For example: ``` val testRelation = LocalRelation.fromExternalRows(Seq("a".attr.int, "b".attr.int, "c".attr.int), 1.to(10).map(_ => Row(1, 2, 3)) ) val query = GlobalLimit(100, testRelation) ``` We can remove this `GlobalLimit`. ### Why are the changes needed? Further optimize the query. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. Closes #31750 from wangyum/SPARK-34628. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This pr remove
GlobalLimitoperator if its child max rows not larger than limit number. For example:We can remove this
GlobalLimit.Why are the changes needed?
Further optimize the query.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.