-
Notifications
You must be signed in to change notification settings - Fork 599
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(batch): parallelize exchange source creation #7125
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we move the creation to do_execute
, so that the Vec
isn't needed 🤔
If your |
I mean eliminating the |
I think I have got your idea. Let me do some refactor. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, but how can we test the effect? Under which circumstances will v
have many partitions? 🤪
Codecov Report
@@ Coverage Diff @@
## main #7125 +/- ##
==========================================
- Coverage 73.20% 73.20% -0.01%
==========================================
Files 1051 1051
Lines 167464 167467 +3
==========================================
Hits 122588 122588
- Misses 44876 44879 +3
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
I have already tested under 3-cn scenarios. One table could have 12 partitions. Previously we will create sources of the exchange operator by RPCs sequentially. This PR can reduce latency for sql like |
I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
select * from v limit 1
ifv
has so many partitions.Checklist
./risedev check
(or alias,./risedev c
)Documentation
If your pull request contains user-facing changes, please specify the types of the changes, and create a release note. Otherwise, please feel free to remove this section.
Types of user-facing changes
Please keep the types that apply to your changes, and remove those that do not apply.
Release note
Please create a release note for your changes. In the release note, focus on the impact on users, and mention the environment or conditions where the impact may occur.
Refer to a related PR or issue link (optional)