disttask: add metrics collection for dispatcher and scheduler

[JK1Zhang]

Enhancement

add metric for dispatcher and scheduler, eg: how many tasks are scheduled, scheduling

Grafana

Dispatcher

• A line chart showing the change in the number of waiting/running/cancelling/reverting tasks over time
• A line chart showing the waiting time of waiting tasks
• A line chart showing the dispatch time of dispatching tasks

Runtime - Scheduler SubTask

• A line chart showing the change in the number of waiting subTasks over time
• A line chart showing the waiting time of waiting subTasks
• A line chart showing the run time of running subTasks

TiDB perspective

• A pie chart showing the distribution of all current subTasks on various TiDB nodes

Task perspective

• A line chart showing the change in the number of each Task's (uncompleted/completed) subTasks over time
• A line chart showing the average rate of each Task (subTask count/hour, which can later be improved to rows/s or bytes/s)

SubTask perspective

• A line chart showing the average running speed of subTasks on different TiDB nodes (subTask count/hour)

Metrics

• dispatch(DDL owner)(for each task, lable: "task_type"=xxxx)
  • Waiting tasks (lable: "status"="waiting")
    • How many
    • Duration (lable: "task_id"=xxxx)
  • Dispatching tasks (lable: "status"="dispatching")
    • Duration (lable: "task_id"=xxxx)
  • Running tasks (lable: "status"="running")
  • Completed tasks (lable: "status"="completed")
  • Reverting tasks (lable: "status"="reverting")
  • Canceling tasks (lable: "status"="canceling")
• Scheduler
  • For each scheduler/each TiDB node (lable: "task_id"=xxxx, "task_type"=xxxx, "scheduler_id"=xxxx, "tidb_id"=xxxx)
    • Running subTask (lable: "status"="running")
      • Duration (lable: "subtask_id"=xxxx)
    • Waiting subTask (lable: "status"="waiting")
      • How many
      • Duration (lable: "subtask_id"=xxxx)