[SPARK-33621][SPARK-33784][SQL][3.1] Add a way to inject data source rewrite rules #30917
Conversation
This PR tries to rename `dataSourceRewriteRules` into something more generic. These changes are needed to address the post-review discussion [here](apache#30558 (comment)). Yes but the changes haven't been released yet. Existing tests. Closes apache#30808 from aokolnychyi/spark-33784. Authored-by: Anton Okolnychyi <aokolnychyi@apple.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
|
@cloud-fan @dongjoon-hyun, here is the backport of PR #30808. |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #133353 has finished for PR 30917 at commit
|
dongjoon-hyun
changed the title
dongjoon-hyun
changed the title
There was a problem hiding this comment.
Hi, @aokolnychyi and all reviewers.
Please note that this includes two JIRAs, but it's technically one PR and its follow-up. Technically, SPARK-33784 is a blocker for Apache Spark 3.1.0.
- [SPARK-33621][SQL] Add a way to inject data source rewrite rules
- [SPARK-33784][SQL] Rename dataSourceRewriteRules batch
I revised the PR title and PR description according to the PR content.
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM.
Merged to branch-3.1 for Apache Spark 3.1.0.
Thank you, @aokolnychyi . Merry Christmas and Happy New Year!
…rewrite rules ### What changes were proposed in this pull request? This PR adds a way to inject data source rewrite rules to branch-3.1 via backporting two JIRA issues. - [SPARK-33621][SQL] Add a way to inject data source rewrite rules - [SPARK-33784][SQL] Rename dataSourceRewriteRules batch ### Why are the changes needed? Right now `SparkSessionExtensions` allow us to inject optimization rules but they are added to operator optimization batch. There are cases when users need to run rules after the operator optimization batch (e.g. cases when a rule relies on the fact that expressions have been optimized). Currently, this is not possible. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? This PR comes with a new test. Closes #30917 from aokolnychyi/backport-spark-33784. Authored-by: Anton Okolnychyi <aokolnychyi@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
HyukjinKwon
left a comment
HyukjinKwon
left a comment
There was a problem hiding this comment.
Yeah, I think this is fine. LGTM
|
Thanks @HyukjinKwon @dongjoon-hyun! I wasn't sure what is the normal procedure for such cherry-picks in Spark so thanks for adapting the PR description, @dongjoon-hyun ! Happy holidays! |
4 participants
What changes were proposed in this pull request?
This PR adds a way to inject data source rewrite rules to branch-3.1 via backporting two JIRA issues.
Why are the changes needed?
Right now
SparkSessionExtensionsallow us to inject optimization rules but they are added to operator optimization batch. There are cases when users need to run rules after the operator optimization batch (e.g. cases when a rule relies on the fact that expressions have been optimized). Currently, this is not possible.Does this PR introduce any user-facing change?
Yes.
How was this patch tested?
This PR comes with a new test.