Add Kubernetes configmap reload to all components

apache · Sep 2, 2022 · 6402fc4 · 6402fc4
1 parent d0f5e7e
commit 6402fc4
Show file tree

Hide file tree

Showing 20 changed files with 284 additions and 113 deletions.
diff --git a/docs/docs/en/architecture/configuration.md b/docs/docs/en/architecture/configuration.md
@@ -17,14 +17,14 @@ The directory structure of DolphinScheduler is as follows:
 │
 ├── licenses directory of licenses
 │
-├── bin directory of DolphinScheduler application commands, configrations scripts 
+├── bin directory of DolphinScheduler application commands, configrations scripts
 │   ├── dolphinscheduler-daemon.sh script to start or shut down DolphinScheduler application
 │   ├── env directory of scripts to load environment variables
 │   │   ├── dolphinscheduler_env.sh script to export environment variables [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...] when you start or stop service using script `dolphinscheduler-daemon.sh`
 │   │   └── install_env.sh script to export environment variables for DolphinScheduler installation when you use scripts `install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`
 │   ├── install.sh script to auto-setup services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
-│   ├── remove-zk-node.sh script to cleanup ZooKeeper caches 
-│   ├── scp-hosts.sh script to copy installation files to target hosts 
+│   ├── remove-zk-node.sh script to cleanup ZooKeeper caches
+│   ├── scp-hosts.sh script to copy installation files to target hosts
 │   ├── start-all.sh script to start all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
 │   ├── status-all.sh script to check the status of all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
 │   └── stop-all.sh script to shut down all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
@@ -34,7 +34,8 @@ The directory structure of DolphinScheduler is as follows:
 │   │   └── start.sh script to start DolphinScheduler alert-server
 │   ├── conf
 │   │   ├── application.yaml configurations of alert-server
-│   │   ├── common.properties configurations of common-service like storage, credentials, etc. 
+│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
+│   │   ├── common.properties configurations of common-service like storage, credentials, etc.
 │   │   ├── dolphinscheduler_env.sh script to load environment variables for alert-server
 │   │   └── logback-spring.xml configurations of alert-service log
 │   └── libs directory of alert-server libs
@@ -44,17 +45,19 @@ The directory structure of DolphinScheduler is as follows:
 │   │   └── start.sh script to start DolphinScheduler api-server
 │   ├── conf
 │   │   ├── application.yaml configurations of api-server
+│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
 │   │   ├── common.properties configurations of common-service like storage, credentials, etc.
 │   │   ├── dolphinscheduler_env.sh script to load environment variables for api-server
 │   │   └── logback-spring.xml configurations of api-service log
 │   ├── libs directory of api-server libs
-│   └── ui directory of api-server related front-end web resources 
+│   └── ui directory of api-server related front-end web resources
 │
 ├── master-server directory of DolphinScheduler master-server commands, configrations scripts and libs
-│   ├── bin 
+│   ├── bin
 │   │   └── start.sh script to start DolphinScheduler master-server
 │   ├── conf
 │   │   ├── application.yaml configurations of master-server
+│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
 │   │   ├── common.properties configurations of common-service like storage, credentials, etc.
 │   │   ├── dolphinscheduler_env.sh script to load environment variables for master-server
 │   │   └── logback-spring.xml configurations of master-service log
@@ -65,13 +68,14 @@ The directory structure of DolphinScheduler is as follows:
 │   │   └── start.sh script to start DolphinScheduler standalone-server
 │   ├── conf
 │   │   ├── application.yaml configurations of standalone-server
+│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
 │   │   ├── common.properties configurations of common-service like storage, credentials, etc.
 │   │   ├── dolphinscheduler_env.sh script to load environment variables for standalone-server
 │   │   ├── logback-spring.xml configurations of standalone-service log
 │   │   └── sql .sql files to create or upgrade DolphinScheduler metadata
 │   ├── libs directory of standalone-server libs
 │   └── ui directory of standalone-server related front-end web resources
-│   
+│  
 ├── tools directory of DolphinScheduler metadata tools commands, configrations scripts and libs
 │   ├── bin
 │   │   └── upgrade-schema.sh script to initialize or upgrade DolphinScheduler metadata
@@ -80,12 +84,13 @@ The directory structure of DolphinScheduler is as follows:
 │   │   └── common.properties configurations of common-service like storage, credentials, etc.
 │   ├── libs directory of tool libs
 │   └── sql .sql files to create or upgrade DolphinScheduler metadata
-│   
+│  
 ├── worker-server directory of DolphinScheduler worker-server commands, configrations scripts and libs
 │ ├── bin
 │ │   └── start.sh script to start DolphinScheduler worker-server
 │ ├── conf
 │ │   ├── application.yaml configurations of worker-server
+│ │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
 │ │   ├── common.properties configurations of common-service like storage, credentials, etc.
 │ │   ├── dolphinscheduler_env.sh script to load environment variables for worker-server
 │ │   └── logback-spring.xml configurations of worker-service log
@@ -107,14 +112,14 @@ Currently, DolphinScheduler just makes a basic config, remember to config furthe
 Default simplified parameters are:
 ```bash
 export DOLPHINSCHEDULER_OPTS="
--server 
--Xmx16g 
--Xms1g 
--Xss512k 
--XX:+UseConcMarkSweepGC 
--XX:+CMSParallelRemarkEnabled 
--XX:+UseFastAccessorMethods 
--XX:+UseCMSInitiatingOccupancyOnly 
+-server
+-Xmx16g
+-Xms1g
+-Xss512k
+-XX:+UseConcMarkSweepGC
+-XX:+CMSParallelRemarkEnabled
+-XX:+UseFastAccessorMethods
+-XX:+UseCMSInitiatingOccupancyOnly
 -XX:CMSInitiatingOccupancyFraction=70
 "
 ```

diff --git a/docs/docs/zh/architecture/configuration.md b/docs/docs/zh/architecture/configuration.md
@@ -13,13 +13,13 @@ DolphinScheduler的目录结构如下：
 │
 ├── licenses licenses存放目录
 │
-├── bin DolphinScheduler命令和环境变量配置存放目录 
+├── bin DolphinScheduler命令和环境变量配置存放目录
 │   ├── dolphinscheduler-daemon.sh 启动/关闭DolphinScheduler服务脚本
 │   ├── env 环境变量配置存放目录
-│   │   ├── dolphinscheduler_env.sh 当使用`dolphinscheduler-daemon.sh`脚本起停服务时，运行此脚本加载环境变量配置文件 [如：JAVA_HOME,HADOOP_HOME, HIVE_HOME ...] 
-│   │   └── install_env.sh 当使用`install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`脚本时，运行此脚本为DolphinScheduler安装加载环境变量配置 
-│   ├── install.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本自动安装服务 
-│   ├── remove-zk-node.sh 清理zookeeper缓存文件脚本 
+│   │   ├── dolphinscheduler_env.sh 当使用`dolphinscheduler-daemon.sh`脚本起停服务时，运行此脚本加载环境变量配置文件 [如：JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
+│   │   └── install_env.sh 当使用`install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`脚本时，运行此脚本为DolphinScheduler安装加载环境变量配置
+│   ├── install.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本自动安装服务
+│   ├── remove-zk-node.sh 清理zookeeper缓存文件脚本
 │   ├── scp-hosts.sh 安装文件传输脚本
 │   ├── start-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本启动所有服务
 │   ├── status-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本获取所有服务状态
@@ -30,7 +30,8 @@ DolphinScheduler的目录结构如下：
 │   │   └── start.sh DolphinScheduler alert-server启动脚本
 │   ├── conf
 │   │   ├── application.yaml alert-server配置文件
-│   │   ├── common.properties 公共服务（存储等信息）配置文件 
+│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
+│   │   ├── common.properties 公共服务（存储等信息）配置文件
 │   │   ├── dolphinscheduler_env.sh alert-server环境变量配置加载脚本
 │   │   └── logback-spring.xml alert-service日志配置文件
 │   └── libs alert-server依赖jar包存放目录
@@ -40,17 +41,19 @@ DolphinScheduler的目录结构如下：
 │   │   └── start.sh DolphinScheduler api-server启动脚本
 │   ├── conf
 │   │   ├── application.yaml api-server配置文件
+│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
 │   │   ├── common.properties 公共服务（存储等信息）配置文件
 │   │   ├── dolphinscheduler_env.sh api-server环境变量配置加载脚本
 │   │   └── logback-spring.xml api-service日志配置文件
 │   ├── libs api-server依赖jar包存放目录
-│   └── ui api-server相关前端WEB资源存放目录 
+│   └── ui api-server相关前端WEB资源存放目录
 │
 ├── master-server DolphinScheduler master-server命令、配置和依赖存放目录
-│   ├── bin 
+│   ├── bin
 │   │   └── start.sh DolphinScheduler master-server启动脚本
 │   ├── conf
 │   │   ├── application.yaml master-server配置文件
+│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
 │   │   ├── common.properties 公共服务（存储等信息）配置文件
 │   │   ├── dolphinscheduler_env.sh master-server环境变量配置加载脚本
 │   │   └── logback-spring.xml master-service日志配置文件
@@ -61,13 +64,14 @@ DolphinScheduler的目录结构如下：
 │   │   └── start.sh DolphinScheduler standalone-server启动脚本
 │   ├── conf
 │   │   ├── application.yaml standalone-server配置文件
+│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
 │   │   ├── common.properties 公共服务（存储等信息）配置文件
 │   │   ├── dolphinscheduler_env.sh standalone-server环境变量配置加载脚本
 │   │   ├── logback-spring.xml standalone-service日志配置文件
 │   │   └── sql DolphinScheduler元数据创建/升级sql文件
 │   ├── libs standalone-server依赖jar包存放目录
 │   └── ui standalone-server相关前端WEB资源存放目录
-│   
+│  
 ├── tools DolphinScheduler元数据工具命令、配置和依赖存放目录
 │   ├── bin
 │   │   └── upgrade-schema.sh DolphinScheduler元数据创建/升级脚本
@@ -76,12 +80,13 @@ DolphinScheduler的目录结构如下：
 │   │   └── common.properties 公共服务（存储等信息）配置文件
 │   ├── libs 元数据工具依赖jar包存放目录
 │   └── sql DolphinScheduler元数据创建/升级sql文件
-│   
+│  
 ├── worker-server DolphinScheduler worker-server命令、配置和依赖存放目录
 │ ├── bin
 │ │   └── start.sh DolphinScheduler worker-server启动脚本
 │ ├── conf
 │ │   ├── application.yaml worker-server配置文件
+│ │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
 │ │   ├── common.properties 公共服务（存储等信息）配置文件
 │ │   ├── dolphinscheduler_env.sh worker-server环境变量配置加载脚本
 │ │   └── logback-spring.xml worker-service日志配置文件
@@ -93,21 +98,21 @@ DolphinScheduler的目录结构如下：
 # 配置文件详解
 
 ## dolphinscheduler-daemon.sh [启动/关闭DolphinScheduler服务脚本]
-dolphinscheduler-daemon.sh脚本负责DolphinScheduler的启动&关闭. 
+dolphinscheduler-daemon.sh脚本负责DolphinScheduler的启动&关闭.
 start-all.sh/stop-all.sh最终也是通过dolphinscheduler-daemon.sh对集群进行启动/关闭操作.
 目前DolphinScheduler只是做了一个基本的设置,JVM参数请根据各自资源的实际情况自行设置.
 
 默认简化参数如下:
 ```bash
 export DOLPHINSCHEDULER_OPTS="
--server 
--Xmx16g 
--Xms1g 
--Xss512k 
--XX:+UseConcMarkSweepGC 
--XX:+CMSParallelRemarkEnabled 
--XX:+UseFastAccessorMethods 
--XX:+UseCMSInitiatingOccupancyOnly 
+-server
+-Xmx16g
+-Xms1g
+-Xss512k
+-XX:+UseConcMarkSweepGC
+-XX:+CMSParallelRemarkEnabled
+-XX:+UseFastAccessorMethods
+-XX:+UseCMSInitiatingOccupancyOnly
 -XX:CMSInitiatingOccupancyFraction=70
 "
 ```
@@ -155,7 +160,7 @@ DolphinScheduler使用Zookeeper进行集群管理、容错、事件监听等功
 
 默认配置如下：
 
-|参数 |默认值| 描述| 
+|参数 |默认值| 描述|
 |--|--|--|
 |registry.zookeeper.namespace|dolphinscheduler|Zookeeper集群使用的namespace|
 |registry.zookeeper.connect-string|localhost:2181| Zookeeper集群连接信息|
@@ -213,7 +218,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置
 
 ## Api-server相关配置
 位置：`api-server/conf/application.yaml`
-|参数 |默认值| 描述| 
+|参数 |默认值| 描述|
 |--|--|--|
 |server.port|12345|api服务通讯端口|
 |server.servlet.session.timeout|120m|session超时时间|
@@ -241,7 +246,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置
 
 ## Master Server相关配置
 位置：`master-server/conf/application.yaml`
-|参数 |默认值| 描述| 
+|参数 |默认值| 描述|
 |--|--|--|
 |master.listen-port|5678|master监听端口|
 |master.fetch-command-num|10|master拉取command数量|
@@ -262,7 +267,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置
 
 ## Worker Server相关配置
 位置：`worker-server/conf/application.yaml`
-|参数 |默认值| 描述| 
+|参数 |默认值| 描述|
 |--|--|--|
 |worker.listen-port|1234|worker监听端口|
 |worker.exec-threads|100|worker工作线程数量,用于限制并行的任务实例数量|
@@ -280,7 +285,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置
 
 ## Alert Server相关配置
 位置：`alert-server/conf/application.yaml`
-|参数 |默认值| 描述| 
+|参数 |默认值| 描述|
 |--|--|--|
 |server.port|50053|Alert Server监听端口|
 |alert.port|50052|alert监听端口|