Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor job manager part1 #3976

Merged
merged 2 commits into from
Mar 10, 2022

Conversation

panda-sheep
Copy link
Contributor

@panda-sheep panda-sheep commented Mar 3, 2022

What type of PR is this?

  • bug
  • feature
  • enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:

refactor job manager, this pr mainly did the following parts

  1. separation by space

  2. Remove job_concurrency
    Because only compact and flush, stats syntax supports job_concurrency, and job passes job_concurrency to task, but the corresponding task does not use job_concurrency.
    Even using job_concurrency does not make much sense, because in a space, compactTask and flushTask are concurrent on the storage according to the number of engines. statsTask is concurrent according to the number of parts. FLAGS_max_concurrent_subtasks is enough

  3. Fix bugs, such as Call thrift RunAdminJob to create a balance job, if space not existed, the job is still created #3743

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

  • Unit test(positive and negative cases)
  • Function test
  • Performance test
  • N/A

Affects:

  • Documentation affected (Please add the label if documentation needs to be modified.)
  • Incompatibility (If it breaks the compatibility, please describe it and add the label.)
  • If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
  • Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

@panda-sheep panda-sheep requested review from CPWstatic, dutor and a team as code owners March 3, 2022 10:15
@panda-sheep panda-sheep requested review from critical27, liuyu85cn and a team March 3, 2022 10:16
Copy link
Contributor

@pengweisong pengweisong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good Job, LGTM

Copy link
Contributor

@critical27 critical27 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally LGTM.

}

JobDescription jobDesc(nebula::value(jobId), cmd, paras);
auto errorCode = jobMgr_->addJob(jobDesc, adminClient_);
Copy link
Contributor

@critical27 critical27 Mar 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The job manager will have one queue for one space later, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the later pr will do the following work:
Each space has a priority queue.
Jobs in the same space are executed serially according to priority.
Jobs in different spaces are executed in parallel according to the space.

Copy link
Contributor

@liuyu85cn liuyu85cn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good job

@critical27 critical27 merged commit 481fef7 into vesoft-inc:master Mar 10, 2022
@panda-sheep panda-sheep deleted the refactor_job_manager branch March 10, 2022 03:32
@Sophie-Xie Sophie-Xie mentioned this pull request Mar 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for review ready-for-testing PR: ready for the CI test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants