resource control inside dist framework

[D3Hunter]

Feature Request

Is your feature request related to a problem? Please describe:

• Currently, the dispatcher manager supports up to 4 tasks, and the Scheduler manager can process up to 4 tasks.
• The dispatcher/Scheduler manager does not consider the concurrent configuration of tasks when processing tasks, and resources may be overused, resulting in OOM.
• The above manager does not consider the order of tasks. After task A is processed halfway, it may start processing other tasks due to failover and other reasons, causing task A to keep waiting.

Describe the feature you'd like:

This feature will add control over cpu/mem/disk within the nodes managed by framework(see tidb_service_scope), improves resource usage, avoids overall resource usage exceeding limits, and ensures task execution order. When executing multiple tasks in parallel, ensure that each task can run normally without affecting each other.
in first version, we consider dist framework can use all resources that it runs on.

resource control

• add fields for resource control #49019
• impl slot manager in dispatcher, and control resource before start dispatcher #49100
• implement scheduler slot manager #49135
• addindex/disttask: adjust add index task concurrency & add check when submit task #49403
• disttask: maintain managed nodes separately #49623
• disttask: consider node load when schedule/balance subtasks #49758
• disttask: Refactor taskExecutor slotManager to handle task occupation #49713
• disttask: init capacity and check concurrency using cpu count of managed node #49875
• import/index: adjust concurrency using cpu-count of managed node depends on use case #50091
• disttask: change extra running subtasks to pending #50266
• disttask: add available resource param to step executor #50475
• add-index restrict resource usage
• import-into restrict resource usage and allow multiple jobs running concurrently
  • importinto: remove separate wait routine in detached mode and refactor #51335
  • importinto: calculate writer mem size using step resource & add a option for test #51486
  • importinto, disttask: allow multi import job & adjust total memory for cgroup v2 #51575
  • importinto: limit global sort minimum thread required #51776
• disttask: alloc or occupy resource when there are subtasks to run & fix scheduler block task #50650
• disttask: do not view normal nodes as background nodes when all background nodes fail #50540
• disttask: decouple handleExecutableTask from manager & separate executor context #50724
• disttask: mock to start multiple scheduler/taskexecutor #50801
• disttask: add integration test for resource control #50962
• disttask: unify task order naming to 'rank' & refactor failpoint  #51136
• disttask: add benchmark to test scheduler overhead #51182
• disttask: extract task/subtask base #51231
• disttask: skip scheduler take slots for some states #51022
• disttask: remove onError #51003
• disttask: rollback subtasks without init taskexecutor #50987
• disttask: enlarge max-concurrent-task and optimize scheduler/executor loop #51241
• disttask: refresh whole task info when state/step changed #51426
• disttask: add metrics for used slots #51662

others

• disttask: rename scheduler to TaskExecutor #49025
• rename old dispatcher to scheduler #49125
• record subtask end time #49617
• disttask: add notify when task submitted or changed & refactor test #49805
• disttask: merge handle task loop #50224
• disttask: remove revert subtask #50550
• disttask: move step2Str to framework #50565
• disttask: replace spool with waitgroup #50664
• disttask: move framework integration tests into separate pkg #50868
• disttask: fix routine leak & refine logs & refactor failpoint #50912
• disttask: refine planErr state transition #51279
• disttask: refine manager retry #51071

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy: