Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(meta): inform executors of the vnodes they own #2887

Merged
merged 6 commits into from
May 30, 2022

Conversation

xx01cyx
Copy link
Contributor

@xx01cyx xx01cyx commented May 29, 2022

What's changed and what's your intention?

Summarize your change

  • Set vnodes into StreamActor proto in meta. Executors in one actor share the same set of vnodes, so in this way, stream executors can know the vnodes they own.
  • Add some comments about the logic of schedule().

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests

Refer to a related PR or issue link

This PR should work with #2882 to enable vnode pruning. Specifically, a stateful executor should use the vnode info it gets from meta in this PR to initialize its keyspace.

@codecov
Copy link

codecov bot commented May 29, 2022

Codecov Report

Merging #2887 (9da70b4) into main (c51c094) will decrease coverage by 0.00%.
The diff coverage is 72.36%.

@@            Coverage Diff             @@
##             main    #2887      +/-   ##
==========================================
- Coverage   72.90%   72.89%   -0.01%     
==========================================
  Files         703      703              
  Lines       93151    93196      +45     
==========================================
+ Hits        67910    67939      +29     
- Misses      25241    25257      +16     
Flag Coverage Δ
rust 72.89% <72.36%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/common/src/hash/dispatcher.rs 91.89% <ø> (ø)
src/meta/src/cluster/mod.rs 59.58% <ø> (ø)
src/meta/src/hummock/level_handler.rs 73.86% <ø> (ø)
src/meta/src/hummock/test_utils.rs 95.80% <ø> (-0.06%) ⬇️
src/meta/src/stream/meta.rs 37.12% <0.00%> (-0.51%) ⬇️
src/meta/src/stream/mod.rs 40.90% <0.00%> (ø)
src/storage/src/hummock/compactor.rs 68.93% <ø> (ø)
src/storage/src/hummock/shared_buffer/mod.rs 85.30% <ø> (ø)
src/storage/src/hummock/sstable/builder.rs 91.52% <ø> (-0.05%) ⬇️
src/storage/src/hummock/sstable/group_builder.rs 93.75% <ø> (ø)
... and 14 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@xx01cyx xx01cyx requested a review from fuyufjh May 30, 2022 01:31
Copy link
Member

@fuyufjh fuyufjh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@xx01cyx xx01cyx enabled auto-merge (squash) May 30, 2022 05:58
@xx01cyx xx01cyx merged commit b6d24f9 into main May 30, 2022
@xx01cyx xx01cyx deleted the cyx/executor-vnode-info branch May 30, 2022 06:02
Enter-tainer pushed a commit that referenced this pull request May 30, 2022
* feat(meta): inform executors of the vnodes they own

* remove maplen field in VNodeBitmap and move it to common.proto

* use bitmap instead of Vec<Vnode>

* fmt proto

* refine code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants