Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Kubernetes configmap reload to all components #11730

Merged
merged 1 commit into from
Sep 2, 2022
Merged

Add Kubernetes configmap reload to all components #11730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions docs/docs/en/architecture/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ The directory structure of DolphinScheduler is as follows:
│
├── licenses directory of licenses
│
├── bin directory of DolphinScheduler application commands, configrations scripts
├── bin directory of DolphinScheduler application commands, configrations scripts
│   ├── dolphinscheduler-daemon.sh script to start or shut down DolphinScheduler application
│   ├── env directory of scripts to load environment variables
│   │   ├── dolphinscheduler_env.sh script to export environment variables [eg: JAVA_HOME,HADOOP_HOME, HIVE_HOME ...] when you start or stop service using script `dolphinscheduler-daemon.sh`
│   │   └── install_env.sh script to export environment variables for DolphinScheduler installation when you use scripts `install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`
│   ├── install.sh script to auto-setup services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│   ├── remove-zk-node.sh script to cleanup ZooKeeper caches
│   ├── scp-hosts.sh script to copy installation files to target hosts
│   ├── remove-zk-node.sh script to cleanup ZooKeeper caches
│   ├── scp-hosts.sh script to copy installation files to target hosts
│   ├── start-all.sh script to start all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│   ├── status-all.sh script to check the status of all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
│   └── stop-all.sh script to shut down all services when you deploy DolphinScheduler in `psuedo-cluster` mode or `cluster` mode
Expand All @@ -34,7 +34,8 @@ The directory structure of DolphinScheduler is as follows:
│   │   └── start.sh script to start DolphinScheduler alert-server
│   ├── conf
│   │   ├── application.yaml configurations of alert-server
│   │   ├── common.properties configurations of common-service like storage, credentials, etc.
│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
│   │   ├── common.properties configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh script to load environment variables for alert-server
│   │   └── logback-spring.xml configurations of alert-service log
│   └── libs directory of alert-server libs
Expand All @@ -44,17 +45,19 @@ The directory structure of DolphinScheduler is as follows:
│   │   └── start.sh script to start DolphinScheduler api-server
│   ├── conf
│   │   ├── application.yaml configurations of api-server
│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
│   │   ├── common.properties configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh script to load environment variables for api-server
│   │   └── logback-spring.xml configurations of api-service log
│   ├── libs directory of api-server libs
│   └── ui directory of api-server related front-end web resources
│   └── ui directory of api-server related front-end web resources
│
├── master-server directory of DolphinScheduler master-server commands, configrations scripts and libs
│   ├── bin
│   ├── bin
│   │   └── start.sh script to start DolphinScheduler master-server
│   ├── conf
│   │   ├── application.yaml configurations of master-server
│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
│   │   ├── common.properties configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh script to load environment variables for master-server
│   │   └── logback-spring.xml configurations of master-service log
Expand All @@ -65,13 +68,14 @@ The directory structure of DolphinScheduler is as follows:
│   │   └── start.sh script to start DolphinScheduler standalone-server
│   ├── conf
│   │   ├── application.yaml configurations of standalone-server
│   │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
│   │   ├── common.properties configurations of common-service like storage, credentials, etc.
│   │   ├── dolphinscheduler_env.sh script to load environment variables for standalone-server
│   │   ├── logback-spring.xml configurations of standalone-service log
│   │   └── sql .sql files to create or upgrade DolphinScheduler metadata
│   ├── libs directory of standalone-server libs
│   └── ui directory of standalone-server related front-end web resources
│  
│  
├── tools directory of DolphinScheduler metadata tools commands, configrations scripts and libs
│   ├── bin
│   │   └── upgrade-schema.sh script to initialize or upgrade DolphinScheduler metadata
Expand All @@ -80,12 +84,13 @@ The directory structure of DolphinScheduler is as follows:
│   │   └── common.properties configurations of common-service like storage, credentials, etc.
│   ├── libs directory of tool libs
│   └── sql .sql files to create or upgrade DolphinScheduler metadata
│  
│  
├── worker-server directory of DolphinScheduler worker-server commands, configrations scripts and libs
│ ├── bin
│ │   └── start.sh script to start DolphinScheduler worker-server
│ ├── conf
│ │   ├── application.yaml configurations of worker-server
│ │   ├── bootstrap.yaml configurations for Spring Cloud bootstrap, mostly you don't need to modify this,
│ │   ├── common.properties configurations of common-service like storage, credentials, etc.
│ │   ├── dolphinscheduler_env.sh script to load environment variables for worker-server
│ │   └── logback-spring.xml configurations of worker-service log
Expand All @@ -107,14 +112,14 @@ Currently, DolphinScheduler just makes a basic config, remember to config furthe
Default simplified parameters are:
```bash
export DOLPHINSCHEDULER_OPTS="
-server
-Xmx16g
-Xms1g
-Xss512k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly
-server
-Xmx16g
-Xms1g
-Xss512k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
"
```
Expand Down
53 changes: 29 additions & 24 deletions docs/docs/zh/architecture/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@ DolphinScheduler的目录结构如下：
│
├── licenses licenses存放目录
│
├── bin DolphinScheduler命令和环境变量配置存放目录
├── bin DolphinScheduler命令和环境变量配置存放目录
│   ├── dolphinscheduler-daemon.sh 启动/关闭DolphinScheduler服务脚本
│   ├── env 环境变量配置存放目录
│   │   ├── dolphinscheduler_env.sh 当使用`dolphinscheduler-daemon.sh`脚本起停服务时，运行此脚本加载环境变量配置文件 [如：JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
│   │   └── install_env.sh 当使用`install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`脚本时，运行此脚本为DolphinScheduler安装加载环境变量配置
│   ├── install.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本自动安装服务
│   ├── remove-zk-node.sh 清理zookeeper缓存文件脚本
│   │   ├── dolphinscheduler_env.sh 当使用`dolphinscheduler-daemon.sh`脚本起停服务时，运行此脚本加载环境变量配置文件 [如：JAVA_HOME,HADOOP_HOME, HIVE_HOME ...]
│   │   └── install_env.sh 当使用`install.sh` `start-all.sh` `stop-all.sh` `status-all.sh`脚本时，运行此脚本为DolphinScheduler安装加载环境变量配置
│   ├── install.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本自动安装服务
│   ├── remove-zk-node.sh 清理zookeeper缓存文件脚本
│   ├── scp-hosts.sh 安装文件传输脚本
│   ├── start-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本启动所有服务
│   ├── status-all.sh 当使用`集群`模式或`伪集群`模式部署DolphinScheduler时，运行此脚本获取所有服务状态
Expand All @@ -30,7 +30,8 @@ DolphinScheduler的目录结构如下：
│   │   └── start.sh DolphinScheduler alert-server启动脚本
│   ├── conf
│   │   ├── application.yaml alert-server配置文件
│   │   ├── common.properties 公共服务（存储等信息）配置文件
│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
│   │   ├── common.properties 公共服务（存储等信息）配置文件
│   │   ├── dolphinscheduler_env.sh alert-server环境变量配置加载脚本
│   │   └── logback-spring.xml alert-service日志配置文件
│   └── libs alert-server依赖jar包存放目录
Expand All @@ -40,17 +41,19 @@ DolphinScheduler的目录结构如下：
│   │   └── start.sh DolphinScheduler api-server启动脚本
│   ├── conf
│   │   ├── application.yaml api-server配置文件
│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
│   │   ├── common.properties 公共服务（存储等信息）配置文件
│   │   ├── dolphinscheduler_env.sh api-server环境变量配置加载脚本
│   │   └── logback-spring.xml api-service日志配置文件
│   ├── libs api-server依赖jar包存放目录
│   └── ui api-server相关前端WEB资源存放目录
│   └── ui api-server相关前端WEB资源存放目录
│
├── master-server DolphinScheduler master-server命令、配置和依赖存放目录
│   ├── bin
│   ├── bin
│   │   └── start.sh DolphinScheduler master-server启动脚本
│   ├── conf
│   │   ├── application.yaml master-server配置文件
│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
│   │   ├── common.properties 公共服务（存储等信息）配置文件
│   │   ├── dolphinscheduler_env.sh master-server环境变量配置加载脚本
│   │   └── logback-spring.xml master-service日志配置文件
Expand All @@ -61,13 +64,14 @@ DolphinScheduler的目录结构如下：
│   │   └── start.sh DolphinScheduler standalone-server启动脚本
│   ├── conf
│   │   ├── application.yaml standalone-server配置文件
│   │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
│   │   ├── common.properties 公共服务（存储等信息）配置文件
│   │   ├── dolphinscheduler_env.sh standalone-server环境变量配置加载脚本
│   │   ├── logback-spring.xml standalone-service日志配置文件
│   │   └── sql DolphinScheduler元数据创建/升级sql文件
│   ├── libs standalone-server依赖jar包存放目录
│   └── ui standalone-server相关前端WEB资源存放目录
│  
│  
├── tools DolphinScheduler元数据工具命令、配置和依赖存放目录
│   ├── bin
│   │   └── upgrade-schema.sh DolphinScheduler元数据创建/升级脚本
Expand All @@ -76,12 +80,13 @@ DolphinScheduler的目录结构如下：
│   │   └── common.properties 公共服务（存储等信息）配置文件
│   ├── libs 元数据工具依赖jar包存放目录
│   └── sql DolphinScheduler元数据创建/升级sql文件
│  
│  
├── worker-server DolphinScheduler worker-server命令、配置和依赖存放目录
│ ├── bin
│ │   └── start.sh DolphinScheduler worker-server启动脚本
│ ├── conf
│ │   ├── application.yaml worker-server配置文件
│ │   ├── bootstrap.yaml Spring Cloud 启动阶段配置文件, 通常不需要修改
│ │   ├── common.properties 公共服务（存储等信息）配置文件
│ │   ├── dolphinscheduler_env.sh worker-server环境变量配置加载脚本
│ │   └── logback-spring.xml worker-service日志配置文件
Expand All @@ -93,21 +98,21 @@ DolphinScheduler的目录结构如下：
# 配置文件详解

## dolphinscheduler-daemon.sh [启动/关闭DolphinScheduler服务脚本]
dolphinscheduler-daemon.sh脚本负责DolphinScheduler的启动&关闭.
dolphinscheduler-daemon.sh脚本负责DolphinScheduler的启动&关闭.
start-all.sh/stop-all.sh最终也是通过dolphinscheduler-daemon.sh对集群进行启动/关闭操作.
目前DolphinScheduler只是做了一个基本的设置,JVM参数请根据各自资源的实际情况自行设置.

默认简化参数如下:
```bash
export DOLPHINSCHEDULER_OPTS="
-server
-Xmx16g
-Xms1g
-Xss512k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly
-server
-Xmx16g
-Xms1g
-Xss512k
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70
"
```
Expand Down Expand Up @@ -155,7 +160,7 @@ DolphinScheduler使用Zookeeper进行集群管理、容错、事件监听等功

默认配置如下：

|参数 |默认值| 描述|
|参数 |默认值| 描述|
|--|--|--|
|registry.zookeeper.namespace|dolphinscheduler|Zookeeper集群使用的namespace|
|registry.zookeeper.connect-string|localhost:2181| Zookeeper集群连接信息|
Expand Down Expand Up @@ -213,7 +218,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置

## Api-server相关配置
位置：`api-server/conf/application.yaml`
|参数 |默认值| 描述|
|参数 |默认值| 描述|
|--|--|--|
|server.port|12345|api服务通讯端口|
|server.servlet.session.timeout|120m|session超时时间|
Expand Down Expand Up @@ -241,7 +246,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置

## Master Server相关配置
位置：`master-server/conf/application.yaml`
|参数 |默认值| 描述|
|参数 |默认值| 描述|
|--|--|--|
|master.listen-port|5678|master监听端口|
|master.fetch-command-num|10|master拉取command数量|
Expand All @@ -262,7 +267,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置

## Worker Server相关配置
位置：`worker-server/conf/application.yaml`
|参数 |默认值| 描述|
|参数 |默认值| 描述|
|--|--|--|
|worker.listen-port|1234|worker监听端口|
|worker.exec-threads|100|worker工作线程数量,用于限制并行的任务实例数量|
Expand All @@ -280,7 +285,7 @@ common.properties配置文件目前主要是配置hadoop/s3/yarn相关的配置

## Alert Server相关配置
位置：`alert-server/conf/application.yaml`
|参数 |默认值| 描述|
|参数 |默认值| 描述|
|--|--|--|
|server.port|50053|Alert Server监听端口|
|alert.port|50052|alert监听端口|
Expand Down
Loading