Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: enforce chain parallelism and fill proper upstream with same vnode range #4740

Merged
merged 15 commits into from
Aug 22, 2022

Conversation

yezizp2012
Copy link
Member

@yezizp2012 yezizp2012 commented Aug 19, 2022

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

As title, follow up of #4710 . Two changes in this PR:

  1. enforce chain parallelism same with its dependent mview.
  2. fill chain actor's upstream with proper actor contains same vnode range, setup vnode bitmap and fragment vnode mapping info when create MoM, rather than when doing scheduling.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

@codecov
Copy link

codecov bot commented Aug 19, 2022

Codecov Report

Merging #4740 (74352d3) into main (0019e59) will decrease coverage by 0.10%.
The diff coverage is 54.25%.

@@            Coverage Diff             @@
##             main    #4740      +/-   ##
==========================================
- Coverage   73.94%   73.83%   -0.11%     
==========================================
  Files         871      871              
  Lines      131548   131846     +298     
==========================================
+ Hits        97267    97346      +79     
- Misses      34281    34500     +219     
Flag Coverage Δ
rust 73.83% <54.25%> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/batch/src/executor/join/lookup_join.rs 56.06% <0.00%> (-0.70%) ⬇️
src/batch/src/task/task_execution.rs 58.46% <ø> (ø)
src/common/src/catalog/mod.rs 70.00% <0.00%> (-2.42%) ⬇️
src/common/src/config.rs 54.01% <ø> (ø)
src/common/src/error.rs 70.53% <ø> (ø)
src/compute/src/rpc/service/stream_service.rs 0.00% <0.00%> (ø)
src/compute/src/server.rs 0.00% <0.00%> (ø)
src/connector/src/source/pulsar/source/reader.rs 0.00% <0.00%> (ø)
src/meta/src/error.rs 26.92% <0.00%> (-3.52%) ⬇️
src/meta/src/manager/catalog/database.rs 34.07% <0.00%> (-0.59%) ⬇️
... and 45 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

src/meta/src/stream/stream_manager.rs Outdated Show resolved Hide resolved
proto/stream_plan.proto Outdated Show resolved Hide resolved
src/meta/src/stream/stream_manager.rs Outdated Show resolved Hide resolved
@yezizp2012 yezizp2012 requested a review from BugenZhao August 22, 2022 06:08
Copy link
Member

@BugenZhao BugenZhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LSTM! 🚀🚀

src/meta/src/stream/mod.rs Outdated Show resolved Hide resolved
src/meta/src/stream/stream_graph.rs Outdated Show resolved Hide resolved
src/meta/src/stream/stream_graph.rs Outdated Show resolved Hide resolved
src/meta/src/stream/stream_manager.rs Outdated Show resolved Hide resolved
src/meta/src/stream/stream_manager.rs Outdated Show resolved Hide resolved
src/meta/src/stream/stream_manager.rs Outdated Show resolved Hide resolved
src/meta/src/stream/stream_manager.rs Show resolved Hide resolved
src/meta/src/stream/stream_manager.rs Outdated Show resolved Hide resolved
proto/stream_plan.proto Outdated Show resolved Hide resolved
@yezizp2012 yezizp2012 requested a review from BugenZhao August 22, 2022 09:04
@mergify mergify bot merged commit dcf7ad2 into main Aug 22, 2022
@mergify mergify bot deleted the feat/chain-parallelism-mapping branch August 22, 2022 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants