Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(meta): decouple steps of stream job preparation (part 1) #7419

Merged
merged 12 commits into from
Jan 18, 2023

Conversation

BugenZhao
Copy link
Member

@BugenZhao BugenZhao commented Jan 16, 2023

I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.

What's changed and what's your intention?

Detailedly explained in #7420. This PR mainly focuses on the internal table handling.

  • Extract the visiting logic into a single function of visit_internal_table.
  • Move the logic of filling IDs before constructing the Fragment Graph, so that once the graph is built, it's complete.
  • Remove a bunch of (mutable) fields in the Context.

Checklist

  • I have written necessary rustdoc comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@@ -791,33 +762,3 @@ where
Ok(id)
}
}

fn get_dependent_relations(fragment_graph: &StreamFragmentGraph) -> MetaResult<Vec<TableId>> {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need this as the frontend provides it now.

@BugenZhao BugenZhao marked this pull request as ready for review January 17, 2023 06:14
…ream-job-steps

Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
@codecov
Copy link

codecov bot commented Jan 17, 2023

Codecov Report

Merging #7419 (0b47542) into main (f0918d7) will increase coverage by 0.08%.
The diff coverage is 69.80%.

@@            Coverage Diff             @@
##             main    #7419      +/-   ##
==========================================
+ Coverage   71.67%   71.75%   +0.08%     
==========================================
  Files        1084     1084              
  Lines      173692   173640      -52     
==========================================
+ Hits       124496   124598     +102     
+ Misses      49196    49042     -154     
Flag Coverage Δ
rust 71.75% <69.80%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/meta/src/manager/streaming_job.rs 18.18% <0.00%> (+18.18%) ⬆️
src/meta/src/rpc/service/ddl_service.rs 0.00% <0.00%> (ø)
src/meta/src/stream/stream_manager.rs 70.06% <50.00%> (ø)
src/meta/src/stream/stream_graph.rs 80.92% <80.62%> (+11.47%) ⬆️
src/meta/src/stream/test_fragmenter.rs 99.30% <100.00%> (+0.01%) ⬆️
src/meta/src/hummock/mock_hummock_meta_client.rs 65.26% <0.00%> (-0.53%) ⬇️
...frontend/src/scheduler/hummock_snapshot_manager.rs 60.00% <0.00%> (-0.48%) ⬇️
src/storage/src/hummock/compactor/mod.rs 83.48% <0.00%> (-0.16%) ⬇️
src/meta/src/hummock/manager/mod.rs 78.17% <0.00%> (-0.07%) ⬇️
... and 7 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Member

@yezizp2012 yezizp2012 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@chenzl25 chenzl25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Copy link
Contributor

@xx01cyx xx01cyx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

src/meta/src/stream/stream_graph.rs Show resolved Hide resolved
@mergify mergify bot merged commit b345a73 into main Jan 18, 2023
@mergify mergify bot deleted the bz/clarify-prepare-stream-job-steps branch January 18, 2023 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants