[Improvement][Deployment] DS should support providing direct deployment solutions on public cloud-managed Kubernetes platform

[SEZ9]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

I strongly support the idea of DS enabling direct deployment on cloud-managed Kubernetes services, particularly Amazon EKS. Here are the reasons:

Growing Cloud Adoption: In today's rapidly evolving cloud computing landscape, over 45% of enterprises in China have adopted public cloud technologies as of 2023, with an annual growth rate of 5%. Managed Kubernetes services are a crucial part of this trend. According to Amazon Web Services, 82% of Kubernetes applications in the cloud run on AWS EKS. As a stable platform, DS should be compatible with such deployment solutions.

Open Source and Compatibility: Open source software is widely embraced because of its neutrality and compatibility, allowing for broad adoption. It should not be limited to supporting only certain open-source services while neglecting cloud-native managed services.

Enhanced Flexibility and Cloud-Native Features: While DS currently offers a Terraform Amazon EC2 deployment solution, this approach lacks the elasticity and full utilization of cloud-native features that an EKS deployment offers. For instance, serverless databases provide better elasticity for DS, and tools like Karpenter enable faster, more efficient scaling.

In conclusion, DS should support deployment across multiple platforms, including public cloud deployment solutions.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[SbloodyS]

DS is an open source component, like Flink and Doris. We only want to support open source k8s. Each public cloud has its own > customized implementation of k8s, and we don't provide deployment solutions for all of them.
These should be implemented by the public cloud itself.

I have already express my my opinion in #16433. I'm -1 on this. Let's see others opinion. @ruanwenjun @Gallardot @EricGao888 @caishunfeng

[Gallardot]

I reviewed this PR #16433 . What specific EKS capabilities is this PR trying to add? karpenter ?

The PR also copies an existing set of Helm charts. Are there any deficiencies with deploying directly using current Helm charts? Why is this new functionality needed? Why not enhance the existing capabilities?

I'm -1 on this.

[SEZ9]

This PR enhances the DS deployment in dolphinscheduler/deploy/terraform/aws. Originally, the deployment was on AWS EC2, but now it supports AWS EKS, with automatic integration of Amazon Aurora Serverless v2 for the database, EFS for persistent storage, S3 for remote logs, and automatic IAM role construction. It also reuses some open-source Kubernetes Helm charts. #16433

[SEZ9]

Here's a simple summary of the key enhancements in this PR:

1. Provides users with a one-click Terraform solution for automatically deploying infrastructure and the DolphinScheduler system on AWS.
2. Integrates key cloud-native features to enhance the robustness of DolphinScheduler's cloud architecture, including managed EKS clusters, Karpenter for elastic scaling, Amazon Aurora Serverless v2 for automatic database scaling, EFS for persistent shared network storage, Amazon S3 for remote logs, dependency and attachment management, and AWS IAM authentication management.
3. Maintains compatibility with the existing DolphinScheduler/deploy/terraform/aws container features.
4. Commits to ongoing maintenance of the modules as DolphinScheduler and AWS services are upgraded.
5. Enhances the convenience and robustness of using DolphinScheduler on the cloud.

[qingwli]

I was -1 on this. My idea is like @SbloodyS

If we want to run ds in eks, we should do some effects on aws part, like they can use some modules or images, like one-click start an eks with ds.

Not change some helm in ds part.

[EricGao888]

IMHO - #16433 (comment)