[DSIP-8][Metrics] Improve DolphinScheduler Monitoring

[EricGao888]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• Monitoring plays an essential role in software stability. However, there is only statics but no metrics in Dolphin Scheduler at present, which means users cannot export metrics to external observation system to monitor their workflows, tasks, as well as DS performance.
• However, to match our slogan Choose good tools, Back home early. Use Right Scheduler, Sleep Tight. we need richer metrics to increase monitoring ability and give our users better experience using Dolphinscheduler, especially in production environment.
• Here are the Email Thread and Proposal.

Use case

• To make the expected improvement described in Description section happen, we could take three steps:

1. List all the metrics we need classified by different parts of Dolphinscheduler, such as master, worker, api server, etc. Here's the doc link for metrics list.
2. Apply the code in the right place and collect these metrics with our metrics-collection frame.
3. Find a method to expose these metrics to external system. related: [Improvement][Common] Use JMX to expose configuration and metrics #5255

Action Items

Stage I

• List the basic metrics for workflow / task / system and embed them in the code: [Feature][metrics] Add master, worker metrics #10326 [Improvement][Metrics] Use tags to indicate task / workflow execution status for metrics #10867
• Enable developers to test and debug metrics conveniently in standalone mode: [Feature][Metrics] Enable prometheus to collect metrics in standalone mode demo #10395
• Establish the naming convention for DS metrics: [Improvement][Metrics] Apply micrometer naming convention to metrics #10432 [Improvement][Metrics] Update some metrics names in grafana-demo dashboards #10552
• Add resource download related metrics for workers: [Feature][Metrics] Add resource download related metrics for workers #10749
• Add metrics for alert server: [Improvement][Metrics] Add metrics for alert server #11131
• Add metrics for api server: [Feature][Metrics] Add metrics for api server #11472
• Check the correctness of metrics when DS deployed with multiple masters and workers.

Stage II

• Make external monitoring system configurable and extensible.
• Add popular exporters supported by Micrometer besides Prometheus, such as CloudWatch, Datadog, StatsD, Influx, JMX, Elastic, etc. For a full list, visit Micrometer Setup section. In addition, to provide users with smooth experience, we should add docker yaml files for each exporter for the demo purpose.

Stage III

• Add user-configurable metrics filter: [Feature][Metrics] Add user-configurable metrics filters #10527
• Increase the granularity and richness of DS metrics to achieve the same or better observability than Apache Airflow: [Feature][Metrics] Increase granularity of metrics #10525

Related issues

related: #5255

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Hi:

• Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.

[SbloodyS]

I think it's better to including the number of threads related to the execution of the worker and master in the monitoring.

[EricGao888]

I just updated the google doc in the Use Case section, taking some metrics into consideration.

Another thing I propose we could think about is the granularity of metrics. I find current metrics are general statistics. Statistics of tasks and workflows are separated. We may need some metric like task.duration.<workflow_id>.<task_id> to monitor vital workflows/tasks more accurately. Of course, a side-effect is we will generate explosive number of metrics, leading to some performance issue. To avoid this, two methods will work:

1. There will be some config for users to switch on/off generating metrics.
2. Dolphin will send those metrics in a UDP way.

[EricGao888]

Besides, we need some descriptions for exiting metrics in official docs. #9441

[ruanwenjun]

@EricGao888 Hi, I close #5255, since there is already a module dolphinscheduler-meter can expose the metrics, and I will take part in this work to provide some common method.

[SbloodyS]

I think this issue is worth DSIP label. WDYT? @zhongjiajie

[EricGao888]

@devosend Hello, may I ask whether it is possible to include the three PRs of stage I in beta-2? In this way, we could get feedback from users in advance and resolve more potential issues before 3.0.0-stable. WDYT

[zhongjiajie]

I think this issue is worth DSIP label. WDYT? @zhongjiajie

Agrees with that, we should add DSIP for this

[zhongjiajie]

@EricGao888 Could you follow the https://dolphinscheduler.apache.org/en-us/community/DSIP.html guide to make it like DSIP?

[zhongjiajie]

@EricGao888 Could you follow the https://dolphinscheduler.apache.org/en-us/community/DSIP.html guide to make it like DSIP?

Oh, I remenber you already discuss with an e-mail about the monitoring in https://lists.apache.org/thread/6sogjh6k7f2hv954mhn24c94l2mzwgsz, maybe you should append some words and tell users we want to covert it to DSIP now

[EricGao888]

Looks like some PRs related to metrics has not been cherry-picked to 3.0.0-prepare. What about picks them when #10867 merged? @ruanwenjun @caishunfeng @zhongjiajie Thx~

[caishunfeng]

Looks like some PRs related to metrics has not been cherry-picked to 3.0.0-prepare. What about picks them when #10867 merged? @ruanwenjun @caishunfeng @zhongjiajie Thx~

I think it's better put into next version, because we are about to release 3.0.0-release, during this time, we only hope to cherry-pick the pr of bugfix.

[EricGao888]

Looks like some PRs related to metrics has not been cherry-picked to 3.0.0-prepare. What about picks them when #10867 merged? @ruanwenjun @caishunfeng @zhongjiajie Thx~

I think it's better put into next version, because we are about to release 3.0.0-release, during this time, we only hope to cherry-pick the pr of bugfix.

Sure, make sense to me. Thx~