Add Kubernetes configmap reload to all components #11730

kezhenxu94 · 2022-09-01T07:20:41Z

[INFO] 2022-09-01 07:15:00.365 +0000 org.apache.dolphinscheduler.server.worker.task.WorkerHeartBeatTask:[86] - [WorkflowInstance-0][TaskInstance-0] - Success write worker group heartBeatInfo into registry, workGroupPath: [/nodes/worker/default/demo-deployment-75b5f8457-ccnck:1234] workerHeartBeatInfo: {"startupTime":1662016367345,"reportTime":1662016500362,"cpuUsage":0.04,"memoryUsage":0.0,"loadAverage":0.17,"availablePhysicalMemorySize":12.96,"maxCpuloadAvg":8.0,"reservedMemory":0.3,"serverStatus":0,"processId":8,"workerHostWeight":100,"workerWaitingTaskCount":0,"workerExecThreadCount":10}
[WARN] 2022-09-01 07:15:03.775 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-kubernetes' not present in namespace : 'default'
[WARN] 2022-09-01 07:15:03.780 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-h2' not present in namespace : 'default'
[INFO] 2022-09-01 07:15:03.781 +0000 org.springframework.cloud.kubernetes.fabric8.config.reload.EventBasedConfigMapChangeDetector:[137] - [WorkflowInstance-0][TaskInstance-0] - Detected change in config maps
[INFO] 2022-09-01 07:15:03.781 +0000 org.springframework.cloud.kubernetes.fabric8.config.reload.EventBasedConfigMapChangeDetector:[61] - [WorkflowInstance-0][TaskInstance-0] - Reloading using strategy: REFRESH
[WARN] 2022-09-01 07:15:04.003 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-kubernetes' not present in namespace : 'default'
[WARN] 2022-09-01 07:15:04.010 +0000 org.springframework.cloud.kubernetes.fabric8.config.Fabric8ConfigUtils:[121] - [WorkflowInstance-0][TaskInstance-0] - config-map with name : 'standalone-server-h2' not present in namespace : 'default'
[INFO] 2022-09-01 07:15:04.011 +0000 org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration:[109] - [WorkflowInstance-0][TaskInstance-0] - Located property source: [BootstrapPropertySource {name='bootstrapProperties-configmap.standalone-server.default'}]
[INFO] 2022-09-01 07:15:04.017 +0000 org.springframework.boot.SpringApplication:[640] - [WorkflowInstance-0][TaskInstance-0] - The following 2 profiles are active: "kubernetes", "h2"
[INFO] 2022-09-01 07:15:04.050 +0000 org.springframework.boot.SpringApplication:[61] - [WorkflowInstance-0][TaskInstance-0] - Started application in 0.261 seconds (JVM running for 152.919)
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[147] - [WorkflowInstance-0][TaskInstance-0] - Master config: listenPort -> 5678 
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[148] - [WorkflowInstance-0][TaskInstance-0] - Master config: fetchCommandNum -> 10 
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[149] - [WorkflowInstance-0][TaskInstance-0] - Master config: preExecThreads -> 10 
[INFO] 2022-09-01 07:15:04.070 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[150] - [WorkflowInstance-0][TaskInstance-0] - Master config: execThreads -> 10 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[151] - [WorkflowInstance-0][TaskInstance-0] - Master config: dispatchTaskNumber -> 3 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[152] - [WorkflowInstance-0][TaskInstance-0] - Master config: hostSelector -> LOWER_WEIGHT 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[153] - [WorkflowInstance-0][TaskInstance-0] - Master config: heartbeatInterval -> PT10S 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[154] - [WorkflowInstance-0][TaskInstance-0] - Master config: taskCommitRetryTimes -> 5 
[INFO] 2022-09-01 07:15:04.071 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[155] - [WorkflowInstance-0][TaskInstance-0] - Master config: taskCommitInterval -> PT1S 
[INFO] 2022-09-01 07:15:04.072 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[156] - [WorkflowInstance-0][TaskInstance-0] - Master config: stateWheelInterval -> PT5S 
[INFO] 2022-09-01 07:15:04.072 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[157] - [WorkflowInstance-0][TaskInstance-0] - Master config: maxCpuLoadAvg -> 8.0 
[INFO] 2022-09-01 07:15:04.072 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[158] - [WorkflowInstance-0][TaskInstance-0] - Master config: reservedMemory -> 0.3 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[159] - [WorkflowInstance-0][TaskInstance-0] - Master config: failoverInterval -> PT10M 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[160] - [WorkflowInstance-0][TaskInstance-0] - Master config: killYarnJobWhenTaskFailover -> true 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[161] - [WorkflowInstance-0][TaskInstance-0] - Master config: registryDisconnectStrategy -> ConnectStrategyProperties(strategy=STOP, maxWaitingTime=PT0S) 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[162] - [WorkflowInstance-0][TaskInstance-0] - Master config: masterAddress -> demo-deployment-75b5f8457-ccnck:5678 
[INFO] 2022-09-01 07:15:04.073 +0000 org.apache.dolphinscheduler.server.master.config.MasterConfig:[163] - [WorkflowInstance-0][TaskInstance-0] - Master config: masterRegistryPath -> /nodes/master/demo-deployment-75b5f8457-ccnck:5678 
[INFO] 2022-09-01 07:15:04.254 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[100] - [WorkflowInstance-0][TaskInstance-0] - Worker config: listenPort -> 1234
[INFO] 2022-09-01 07:15:04.255 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[101] - [WorkflowInstance-0][TaskInstance-0] - Worker config: execThreads -> 20
[INFO] 2022-09-01 07:15:04.256 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[102] - [WorkflowInstance-0][TaskInstance-0] - Worker config: heartbeatInterval -> PT10S
[INFO] 2022-09-01 07:15:04.257 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[103] - [WorkflowInstance-0][TaskInstance-0] - Worker config: hostWeight -> 100
[INFO] 2022-09-01 07:15:04.258 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[104] - [WorkflowInstance-0][TaskInstance-0] - Worker config: tenantAutoCreate -> true
[INFO] 2022-09-01 07:15:04.259 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[105] - [WorkflowInstance-0][TaskInstance-0] - Worker config: tenantDistributedUser -> false
[INFO] 2022-09-01 07:15:04.259 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[106] - [WorkflowInstance-0][TaskInstance-0] - Worker config: maxCpuLoadAvg -> 8
[INFO] 2022-09-01 07:15:04.260 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[107] - [WorkflowInstance-0][TaskInstance-0] - Worker config: reservedMemory -> 0.3
[INFO] 2022-09-01 07:15:04.260 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[108] - [WorkflowInstance-0][TaskInstance-0] - Worker config: groups -> [default]
[INFO] 2022-09-01 07:15:04.260 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[109] - [WorkflowInstance-0][TaskInstance-0] - Worker config: alertListenHost -> localhost
[INFO] 2022-09-01 07:15:04.261 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[110] - [WorkflowInstance-0][TaskInstance-0] - Worker config: alertListenPort -> 50052
[INFO] 2022-09-01 07:15:04.261 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[111] - [WorkflowInstance-0][TaskInstance-0] - Worker config: registryDisconnectStrategy -> ConnectStrategyProperties(strategy=STOP, maxWaitingTime=PT0S)
[INFO] 2022-09-01 07:15:04.262 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[112] - [WorkflowInstance-0][TaskInstance-0] - Worker config: workerAddress -> ConnectStrategyProperties(strategy=STOP, maxWaitingTime=PT0S)
[INFO] 2022-09-01 07:15:04.262 +0000 org.apache.dolphinscheduler.server.worker.config.WorkerConfig:[113] - [WorkflowInstance-0][TaskInstance-0] - Worker config: workerGroupRegistryPaths: [/nodes/worker/default/demo-deployment-75b5f8457-ccnck:1234]
[INFO] 2022-09-01 07:15:09.042 +0000 org.apache.dolphinscheduler.server.master.task.MasterHeartBeatTask:[68] - [WorkflowInstance-0][TaskInstance-0] - Success write master heartBeatInfo into registry, masterRegistryPath: /nodes/master/demo-deployment-75b5f8457-ccnck:5678, heartBeatInfo: {"startupTime":1662016367345,"reportTime":1662016509037,"cpuUsage":0.08,"memoryUsage":0.0,"loadAverage":0.16,"availablePhysicalMemorySize":12.96,"maxCpuloadAvg":8.0,"reservedMemory":0.3,"processId":8}
[INFO] 2022-09-01 07:15:10.371 +0000 org.apache.dolphinscheduler.server.worker.task.WorkerHeartBeatTask:[86] - [WorkflowInstance-0][TaskInstance-0] - Success write worker group heartBeatInfo into registry, workGroupPath: [/nodes/worker/default/demo-deployment-75b5f8457-ccnck:1234] workerHeartBeatInfo: {"startupTime":1662016367345,"reportTime":1662016510366,"cpuUsage":0.02,"memoryUsage":0.0,"loadAverage":0.14,"availablePhysicalMemorySize":12.96,"maxCpuloadAvg":8.0,"reservedMemory":0.3,"serverStatus":0,"processId":8,"workerHostWeight":100,"workerWaitingTaskCount":0,"workerExecThreadCount":20}

As we can see Detected change in config maps and Reloading using strategy: REFRESH, and from the log workerExecThreadCount":10 to workerExecThreadCount":20 when I modify the config map to change the workerExecThreadCount from 10 to 20.

Now all these work out of box in Kubernetes.

kezhenxu94 · 2022-09-01T07:23:32Z

All works as before, when users want to change the configuration, just create a patch config like this, and it's all done.

cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  application.yml: |-
    worker:
      exec-threads: 30
kind: ConfigMap
metadata:
  name: standalone-server
  namespace: default
EOF

codecov-commenter · 2022-09-01T13:32:19Z

Codecov Report

Merging #11730 (f88feec) into dev (d0f5e7e) will decrease coverage by 0.09%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##                dev   #11730      +/-   ##
============================================
- Coverage     39.73%   39.63%   -0.10%     
+ Complexity     4709     4689      -20     
============================================
  Files          1008     1008              
  Lines         37878    37881       +3     
  Branches       4238     4238              
============================================
- Hits          15049    15016      -33     
- Misses        21218    21259      +41     
+ Partials       1611     1606       -5

Impacted Files	Coverage Δ
...inscheduler/server/log/LoggerRequestProcessor.java	`17.30% <0.00%> (-0.52%)`	⬇️
...eduler/server/worker/task/WorkerHeartBeatTask.java	`68.08% <0.00%> (-8.52%)`	⬇️
...erver/master/processor/queue/TaskEventService.java	`75.00% <0.00%> (-5.36%)`	⬇️
...r/plugin/task/sqoop/parameter/SqoopParameters.java	`55.12% <0.00%> (-1.29%)`	⬇️
...ache/dolphinscheduler/plugin/task/k8s/K8sTask.java	`89.47% <0.00%> (ø)`
...lphinscheduler/plugin/task/k8s/K8sTaskChannel.java	`0.00% <0.00%> (ø)`
...lphinscheduler/plugin/task/dq/DataQualityTask.java	`0.00% <0.00%> (ø)`
...heduler/plugin/task/dq/DataQualityTaskChannel.java	`0.00% <0.00%> (ø)`
...inscheduler/plugin/task/k8s/K8sTaskParameters.java
...ler/plugin/task/dq/utils/spark/SparkArgsUtils.java
... and 10 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/resources/bootstrap.yaml

sonarcloud · 2022-09-02T03:22:05Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

kezhenxu94 requested review from caishunfeng, SbloodyS and ruanwenjun as code owners September 1, 2022 07:20

github-actions bot added the backend label Sep 1, 2022

kezhenxu94 force-pushed the config-reload branch from 9bc4217 to f88feec Compare September 1, 2022 12:22

kezhenxu94 requested a review from EricGao888 September 2, 2022 02:07

EricGao888 previously approved these changes Sep 2, 2022

View reviewed changes

EricGao888 reviewed Sep 2, 2022

View reviewed changes

dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/resources/bootstrap.yaml Show resolved Hide resolved

Add Kubernetes configmap reload to all components

6402fc4

kezhenxu94 dismissed EricGao888’s stale review via 6402fc4 September 2, 2022 02:52

kezhenxu94 force-pushed the config-reload branch from f88feec to 6402fc4 Compare September 2, 2022 02:52

kezhenxu94 requested review from zhongjiajie and Tianqi-Dotes as code owners September 2, 2022 02:52

github-actions bot added the document label Sep 2, 2022

EricGao888 approved these changes Sep 2, 2022

View reviewed changes

EricGao888 assigned kezhenxu94 Sep 2, 2022

EricGao888 added this to the 3.1.0 milestone Sep 2, 2022

kezhenxu94 merged commit 277f137 into dev Sep 2, 2022

kezhenxu94 deleted the config-reload branch September 2, 2022 04:03

kezhenxu94 mentioned this pull request Sep 5, 2022

[Improvement-11505][k8s] Use configmap to mount the configuration files of each server #11508

Closed

caishunfeng mentioned this pull request Sep 8, 2022

[DSIP-6][k8s] k8s on dolphinscheduler #9312

Closed

1 task

fengjian1129 pushed a commit to fengjian1129/dolphinscheduler that referenced this pull request Sep 13, 2022

Add Kubernetes configmap reload to all components (apache#11730)

6869c50

fengjian1129 pushed a commit to fengjian1129/dolphinscheduler that referenced this pull request Sep 29, 2022

Add Kubernetes configmap reload to all components (apache#11730)

a8c0073

xdu-chenrj pushed a commit to xdu-chenrj/dolphinscheduler that referenced this pull request Oct 13, 2022

Add Kubernetes configmap reload to all components (apache#11730)

fbe6eda

kezhenxu94 mentioned this pull request Oct 27, 2022

Add configmap resource permissions so config hot reload can work #12572

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Kubernetes configmap reload to all components #11730

Add Kubernetes configmap reload to all components #11730

kezhenxu94 commented Sep 1, 2022

kezhenxu94 commented Sep 1, 2022

codecov-commenter commented Sep 1, 2022 •

edited

Loading

sonarcloud bot commented Sep 2, 2022

Add Kubernetes configmap reload to all components #11730

Add Kubernetes configmap reload to all components #11730

Conversation

kezhenxu94 commented Sep 1, 2022

kezhenxu94 commented Sep 1, 2022

codecov-commenter commented Sep 1, 2022 • edited Loading

Codecov Report

sonarcloud bot commented Sep 2, 2022

codecov-commenter commented Sep 1, 2022 •

edited

Loading