Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow scaling Materialize along with Chain for created "MV on MV". #5740

Closed
Tracked by #5728
BugenZhao opened this issue Oct 10, 2022 · 0 comments · Fixed by #6168
Closed
Tracked by #5728

Allow scaling Materialize along with Chain for created "MV on MV". #5740

BugenZhao opened this issue Oct 10, 2022 · 0 comments · Fixed by #6168
Assignees
Labels
component/streaming Stream processing related issue. type/feature

Comments

@BugenZhao
Copy link
Member

BugenZhao commented Oct 10, 2022

This is the only case of no-shuffle exchange now. To support scaling them simultaneously, we can specify that only upstream Materialize is allowed to be the target of rescheduling.

  • After feat: enforce chain parallelism and fill proper upstream with same vnode range #4740, we're ensured that the parallel units of Chain must be the same as the upstream Materialize, connected with no-shuffle exchange.
  • If we're asked to scale the Materialize, we need to resolve the downstream Chain and clone the plan for it.
  • The resolving is recursive:
    • It's possible the downstream MV is simple enough, so the Chain fragment itself also includes a Materialize. Then we need to check the down-downstream.
    dev=> create table t (v int);
    CREATE_TABLE
    dev=> create materialized view mv1 as select * from t order by v limit 10;
    CREATE_MATERIALIZED_VIEW
    dev=> create materialized view mv2 as select * from mv1;
    CREATE_MATERIALIZED_VIEW
    dev=> create materialized view mv3 as select * from mv2;
    CREATE_MATERIALIZED_VIEW
    
  • Disallow directly scale the fragment containing Chain. So we must at the top when resolving recursively.
  • If the state of any of the table fragments is Creating in the recursive resolving results, we should refuse this request.
    // Fragments of a Materialized View
    message TableFragments {
    // The state of the fragments of this table
    enum State {
    UNSPECIFIED = 0;
    // The materialized view is creating
    CREATING = 1;
    // The materialized view has been created
    CREATED = 2;
    }
@github-actions github-actions bot added this to the release-0.1.14 milestone Oct 10, 2022
@BugenZhao BugenZhao changed the title Allow scaling Materialize along with Chain for created "MV on MV".This is the only case of no-shuffle exchange now. To support scaling them simultaneously, we can specify that only upstream Chain is allowed to be the target of rescheduling. Allow scaling Materialize along with Chain for created "MV on MV". Oct 10, 2022
@BugenZhao BugenZhao added type/feature component/streaming Stream processing related issue. labels Oct 14, 2022
@shanicky shanicky self-assigned this Oct 14, 2022
@shanicky shanicky linked a pull request Nov 2, 2022 that will close this issue
2 tasks
@mergify mergify bot closed this as completed in #6168 Nov 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/streaming Stream processing related issue. type/feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants