[DSIP-66][k8s] YAML Pod for User Customization

[Mighten]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

Supporting user-customized K8s YAML Pod has the following benefits:

• Flexibility: Unlike the existing K8s low-code job with limited functionality, YAML Pod provide users with the flexibility to define sophisticated task instances in DolphinScheduler, similar to how custom JSON does in DataX.

• Workflow Customization: Users can integrate operational and maintenance processes into DolphinScheduler using YAML Pod for complex workflows.

In short, by enabling user-customized YAML Pod, DolphinScheduler can better support a wide range of Kubernetes-based workflows and operational requirements.

Design Detail

Frontend Design

The frontend adds support for user-customized k8s YAML Pod while remaining compatible with the original k8s low-code jobs.

1. The Web UI layouts

   

   When the user switches on the Custom Template, the Low-code k8s Job fields should hide and YAML editor should appear (or vice versa), similar to the JSON Custom Template in the DataX plugin.

   This feature is implemented using the Vue component span, which is controlled by reactive variables (such as yamlEditorSpan) in the file dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts.

2. The Request body

   When the user switches to Custom Template mode, the request body should include only YAML-related fields (customConfig and yamlContent), and all previously hidden fields should not be sent.

   This feature is implemented using the taskParams in the file dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts

3. i18n/locales

   Apache DolphinScheduler is an international software and should support multiple languages.

   The text on the Web UI are retrieved from variables defined in the file dolphinscheduler-ui/src/locales/{en_US, zh_CN}/project.ts. And for user-customized k8s YAML tasks, there are three key variables to consider:

   • k8s_custom_template: the label for the switch to enable user-customized k8s YAML tasks.
   • k8s_yaml_template: the label for the text editor used to input user YAML.
   • k8s_yaml_empty_tips: the warning message displayed when a user tries to submit empty YAML

   This feature is implemented by invoking t('project.node.${variable_name}') (such as t('project.node.k8s_yaml_template')) in the file dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts.

Backend Design

The backend design describes the process of how the worker executes user-customized k8s YAML Pod tasks.

After the worker checks the parameters, K8sYamlTaskExecutor is loaded for the current user-customized k8s YAML Pod task. Once the YAML is parsed into HasMetadata, its kind field is used to assign abstractK8sOperation as K8sPodOperation for executing the YAML Pod task.

1. K8s Task Executors

   

   Three k8s task executor are involved:

   • AbstractK8sTaskExecutor is an abstract class that represents a k8s task executor.
   • K8sTaskExecutor is a concrete class that extends AbstractK8sTaskExecutor to represent a low-code executor
   • K8sYamlTaskExecutor is a concrete class that extends AbstractK8sTaskExecutor to represent a user-customized k8s YAML task executor.

2. K8s Operation handlers

   

   Two operation handlers are involved:

   • AbstractK8sOperation is an interface representing all k8s resource operations.
   • K8sPodOperation is a concrete class that implements AbstractK8sOperation to handle Pod operations

Usecase Design

A typical use case for a k8s YAML Pod task includes uploading YAML, online workflows, and starting workflows, similar to k8s low-code jobs, unless users switch to the Custom Template option to fill in Pod YAML.

1. The user edits a k8s Pod YAML node in a workflow
2. If the Custom Template is activated and YAML content is not blank, the user may online this whole workflow
3. If the workflow is online, the user may start the workflow and review the logs generated during the execution of the workflow.

Compatibility, Deprecation, and Migration Plan

The user-customized k8s YAML feature requires only customConfig to be activated(1). By default, the value is 0, which applies to the existing k8s low-code jobs.

This DSIP does not propose any deprecation or migration plan, but community feedback is valued.

Test Plan

The User-customized k8s YAML Pod feature allows users to submit YAML Pod to k8s. And this test plan aims to ensure that the feature functions as expected and meets user requirements.

Test Case #	Name	Action	Expectation
1	UI Display	Edit YAML, save and reopen	The YAML content stays up-to-date.
2	UI Validation	try to submit empty YAML	The UI modal dialog intercepts empty YAML.
3	Online Workflow	Save workflow, and online	The User successfully brings the workflow online.
4	Dryrun Workflow	Run workflow as dryrun mode	The Master successfully dry runs this task.
5	Test Workflow	Run workflow as test mode	The Worker successfully tests this task.
6	Run Workflow	Run workflow	The Worker successfully runs this task.

Code of Conduct

• I agree to follow this project's Code of Conduct