-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEAT] (ACTORS-2) Add optimization pass to split Project into ActorPoolProject #2627
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2627 +/- ##
=======================================
Coverage ? 63.56%
=======================================
Files ? 982
Lines ? 112330
Branches ? 0
=======================================
Hits ? 71407
Misses ? 40923
Partials ? 0
|
e2b4c54
to
74447f1
Compare
37d8728
to
481a0dd
Compare
bf69252
to
2dd4452
Compare
077d92d
to
1fb4fb0
Compare
772c9a2
to
19dca2d
Compare
Implement core logic Add TreeNodeRewriter logic Add passing unit test Add test with multiple stateful projections Add another test that is currently failing
d0bc513
to
dc7e648
Compare
src/daft-plan/src/logical_optimization/rules/split_actor_pool_projects.rs
Outdated
Show resolved
Hide resolved
src/daft-plan/src/logical_optimization/rules/split_actor_pool_projects.rs
Outdated
Show resolved
Hide resolved
src/daft-plan/src/logical_optimization/rules/split_actor_pool_projects.rs
Show resolved
Hide resolved
…projects.rs Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com>
…projects.rs Co-authored-by: Desmond Cheong <desmondcheongzx@gmail.com>
CodSpeed Performance ReportMerging #2627 will not alter performanceComparing Summary
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome, thanks for adding the test!
Adds logical optimization for splitting a
Project
with stateful UDFs into a chain of(Project -> ActorPool -> ActorPool -> ... -> Project)
We then rely on ProjectionPushdown rules to optimize this chain (applying column pruning, projection fusion etc).