Skip to content

Releases: datavane/datasophon

v1.2.1 release

03 Dec 10:11
@datasophon datasophon
v1.2.1
38a9f1c
Compare
Choose a tag to compare
v1.2.1 release Latest
Latest

新特性：

1、支持自定义HDFS命名空间。

2、新增Flink集成Iceberg1.4.0。

3、升级StreamPark版本至2.1.1。

4、升级Flink版本至1.16.2。

5、新增集成Kyuubi。

6、升级DolphinScheduler版本至3.1.8。

Bug修复：

1、修复NameNode和ResourceManager active与standby状态判断错误问题。

2、修复HDFS开启Kerberos安全认证后，ssl-client.xml和ssl-server.xml生成不正确问题。

3、优化开启Kerberos安全认证后，principal和keytab文件生成逻辑。

4、优化shell命令执行日志打印。

5、优化部分部署安装和组件集成文档。

What's Changed

New Contributors

Full Changelog: v1.2.0...v1.2.1

Contributors

88fantasy, a19920714liou, and 7 other contributors
Assets 3
Loading

v1.2.0 release

19 Sep 16:16
@datasophon datasophon
v1.2.0
9da86d3
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.2.0 release

新特性：

1、升级spring boot版本为2.6.1。

2、新增项目启动时自动创建数据表和初始化数据功能。

3、优化doris部署监控管理，增加doris自动进行高可用集群配置功能，升级doris版本为1.2.6。

4、优化服务指令执行流程，避免出现服务指令进度卡死的情况。

5、新增datasophon-init模块，可通过脚本自动初始化集群环境。

6、优化hive默认使用hive on yarn环境配置。

7、新增集群删除功能。

8、新增多个生产上关心的监控指标，例如HDFS 整个集群的客户端连接数、NameNode 丢失的块数、DataNode RPC 被调用次数、DataNode

RPC 队列积压长度等。

Bug修复：

1、修复不选择安装的服务实例时依然生成服务指令的问题。

2、修复租户管理中租户列表分页不生效的问题。

3、修复服务角色实例警告报警状态无法恢复的问题。

4、修复租户管理租户列表和用户组列表未按集群隔离的问题。

5、修复doris安装部署时，未自动生成doris be数据目录的问题。

6、修复hdfs安装部署时出现的ileNotFoundException: /etc/security/keytab/keystore问题。

7、 修复项目启动时出现的“because it exists, maybe from xml file”错误问题

What's Changed

New Contributors

Full Changelog: v1.1.2...v1.2.0

Contributors

88fantasy, a19920714liou, and 10 other contributors
Assets 3
Loading

v1.1.2

27 Jul 08:27
@datasophon datasophon
v1.1.2
b51798d
Compare
Choose a tag to compare
v1.1.2

New features:

  1. Host management adds functions such as starting host services, stopping workers, and redistributing workers.
  2. Alarm management has added the function of disabling indicators.
  3. Tenant information is isolated according to the cluster in tenant management.
  4. HDFS automatically updates HDFS Web Ui after turning on Kerberos.
  5. In alarm management, the alarm indicators are added with a status to be updated. When updating the alarm indicators, the alarm indicator status is changed.
  6. Add remote loading repository function.

Bug fix:

  1. Fix the issue of the host disappearing from the host management list when the master slave heartbeat fails.
  2. Fix the issue of adding a new host being unable to add a deleted host.
  3. Fix the NullPointException issue in the PrometheusActor Class when adding a host for the first time.
  4. Fix: When installing services on the main node, there is no need to download the installation package to avoid downloading overwriting and causing damage to the source installation package.
  5. Fix the issue of some alarm indicators not being effective.
  6. Fix the issue of incorrect indicators displayed in some monitoring overview.

新特性：

1、主机管理新增启动主机服务、停止Worker、重新分发Worker功能。

2、告警管理新增停用指标功能。

3、租户管理中租户信息按照集群进行隔离。

4、HDFS在开启Kerberos后自动更新HDFS Web Ui。

5、告警管理中告警指标新增待更新状态，在更新告警指标时，变更告警指标状态。

6、新增远程加载存储库功能。

Bug修复：

1、修复主从心跳失败时主机从主机管理列表消失的问题。

2、修复添加新主机无法添加已经删除的主机的问题。

3、修复初次添加主机时，PrometheusActor Class出现的NullPointException异常问题。

4、修复当在主节点安装服务时，无需下载安装包，避免下载覆盖导致源安装包损坏。

5、修复部分告警指标不生效的问题。

6、修复部分监控总览中展示的指标不正确问题。

What's Changed

  • fix issues/277 Prometheus port occupation by @javaht in #280
  • [DS-274][fix] 增加Zookeeper参数单位说明 by @ceohui in #281
  • add sshd eddsa depends by @zhzhenqin in #273
  • fix datasophon-ui compile type error by @javaht in #289
  • [Feature-288][datasophon-api] 集群能授权用户为管理员，同时也能解除授权 by @javaht in #295
  • [Fix-275][api] remove the host from the cache when delete the host by @hzluting in #302
  • fix remove multi-hosts throws exception, host ids can not convert to integer by @zhzhenqin in #294
  • [DS-274][fix] 增加Zookeeper参数单位说明 by @ceohui in #301
  • [Improve]Remove useless files by @liugddx in #311
  • [Improve]Use maven revision to manage project version by @liugddx in #310
  • [Feature] Add flyway and h2 by @liugddx in #308
  • [Feature-293][datasophon-ui]Change the form item "role group list" to… by @javaht in #315
  • [Feature-303][manager,worker] Add tez support, add yarn timeline and hive on tez by @zhzhenqin in #312
  • [Bug-316][manager] The Frame Service(Component) cannot be deleted when it installed #316 by @zhzhenqin in #318
  • [Improvement-320][manager,common] Optimize hosts Management, support host service startup, host agent reinstall by @zhzhenqin in #323
  • [Improve]Adapt to the front-end code, which can be used locally by @liugddx in #321
  • [bugfix] Fix the problem of failure to run in the linux operating system. by @liugddx in #322
  • [Improvement-320][manager] Optimize hosts Management, hostid or hostname compatible by @zhzhenqin in #325
  • [Improvement][common] Remove unnecessary depends，suchas：reload4j by @zhzhenqin in #327
  • [Fix-285][API] NullPointException occurs when creating Prometheus con… by @a19920714liou in #332
  • [improve]Compatible with existing environment by @liugddx in #330
  • [Bugfix]Fix the problem that it cannot start on the linux platform by @liugddx in #328
  • [Feature][API] Support Tenant management list isolated by cluster by @chyueyi in #337
  • [Fix-285][common,api,worker] When the worker and manager are on the s… by @WujieRen in #338
  • [Refactor]refactor test code by @liugddx in #334
  • [Impove] Optimize SQL, create database automatically if it doesn't exist by @green241 in #341
  • Fixed DorisFE errors displayed in prometheus by @a19920714liou in #343
  • [Feature-292] [Alert] Add "Disable indicator" button. by @WujieRen in #347

New Contributors

Full Changelog: v1.1.1...v1.1.2

Contributors

hzluting, a19920714liou, and 7 other contributors
Assets 3
Loading

v1.1.1 release

19 Jun 11:46
@datasophon datasophon
v1.1.1
a6bf6b8
Compare
Choose a tag to compare
v1.1.1 release

New features:

  1. The logs generated by the worker when executing service operation instructions will be written to different log files according to service classification, making it easy to query the logs generated by the service during installation or startup and shutdown processes.
  2. The alarm configuration increases the trigger duration configuration, making it easy to adapt to scenarios that require continuous detection for a period of time to determine the alarm.
  3. Update the ZooKeeper service status detection mechanism and cancel the Jps method to obtain the ZooKeeper process pid.
  4. When deleting a host, synchronously delete the host configuration information on Prometheus.
  5. Kafka has added replica. tag. time. max.ms and num. replica. fetchers configurations to facilitate tuning of replica synchronization strategies under high load in Kafka clusters.

Bug Fix：

  1. Fix the issue of port conflicts when Kafka creates Topic through scripts.
  2. Fix the issue of configuration not updating in individual cases.
  3. Fixed the issue where the user permissions of the installation package directory were not updated in some cases.
  4. Fix the issue of unsuccessful installation of Prometheus and missing collection time configuration in Prometheus.yml in some cases.
  5. Fix the issue of Flink high availability configuration not taking effect.
  6. Fix cluster editing errors.
  7. Fix the issue where the master heap memory usage indicator on the overview monitoring page is empty.
  8. Fix the issue of semicolon characters when creating the Doris BE directory during Doris deployment.

新特性：

1、Worker在执行服务操作指令时产生的日志将按照服务分类写入不同的日志文件中，便于查询服务在安装或启动停止过程中产生的日志。

2、告警配置增加触发时长配置，便于适配需要连续检测一段时间才能确定告警的场景。

3、更新ZooKeeper服务状态检测机制，取消Jps方式获取ZooKeeper进程pid。

4、删除主机时，同步删除在Prometheus上的主机配置信息。

5、Kafka新增replica.lag.time.max.ms 和num.replica.fetchers配置，便于在Kafka集群负载高的情况下，调优副本同步策略。

Bug修复：

1、修复Kafka通过脚本创建Topic出现端口冲突的问题。

2、修复个别情况下配置不更新问题。

3、修复某些情况下安装包目录所属用户权限没有更新的问题。

4、修复某些情况下Prometheus安装不成功，prometheus.yml缺少采集时间配置的问题。

5、修复Flink高可用配置不生效的问题。

6、修复集群编辑出错的问题。

7、修复总览监控页面master堆内存使用率指标为空的问题。

8、修复Doris部署时，创建Doris BE目录出现分号字符的问题。

New Contributors

Full Changelog: v1.1.0...v1.1.1

Contributors

iuhoay, leeoo, and 8 other contributors
Assets 3
Loading

v1.1.1-beta release

24 Apr 16:10
@datasophon datasophon
v1.1.1-beta
0f13686
Compare
Choose a tag to compare
v1.1.1-beta release

#New features:

  1. After restarting the server operating system, the service will start automatically upon startup.
  2. Upgrade hbase to version 2.4.16.
  3. HDFS namenode and Yarn ResourceManager WebUi have added master-slave identities.
  4. The default number of built-in topic replicas in KAFKA has been changed to 3.
  5. HDFS added Block write rate and Rpc processing time trend monitoring chart.
  6. Add heap memory parameter configuration to the namenode and datanode of HDFS.
  7. Host management adds IP condition search.
  8. Added support for integrating Doris high-performance analysis databases.
  9. Kerberos can not be deployed on the same machine as DataSophon Web.
  10. Added support for web front-end and back-end merge deployment, eliminating the need for separate deployment of nginx proxy front-end.

#Bug fix:

  1. Fixed the issue of not updating the configuration after adding a new client instance. The expected result is to continue using the default role group configuration, but the system will automatically add a rolegroup1 role group.
  2. After fixing the cluster machine/service instance scaling, the machine quantity indicator on the overview page was not changed accordingly.
  3. Fix the keytab file acquisition failure of the multi cluster management service, which resulted in the service being unable to start.
  4. Repair the synchronization of deleting corresponding alarms when deleting services.
  5. Fix the issue of incorrect cluster ID parameter transmission in a multi cluster environment.
  6. Fixed a bug in the semi transparent display of background command lists after multiple pages.
  7. Fixed a bug where global variable acquisition failed in a multi cluster environment.
  8. Optimize the DataSophon web service startup script and remove some memory parameters.
  9. Fix the issue of occasional failure to obtain local Actors.

新特性：

1、服务器操作系统重启后服务实现开机自启。

2、hbase升级为2.4.16版本。

3、HDFS namenode、yarn ResourceManager WebUi新增主从标识。

4、 KAFKA内置topic副本数默认改为3。

5、HDFS新增Block写入速率，Rpc处理耗时趋势监控图表。

6、HDFS的namenode、datanode增加堆内存参数配置。

7、主机管理新增ip条件搜索。

8、新增Worker停止时，同时停止Node Exporter。

9、新增支持集成Doris高性能分析数据库。

10、Kerberos可以不和DataSophon Web部署在同一台机器上。

11、新增支持Web前后端合并部署，不需再单独部署nginx代理前端。

Bug修复：

1、修复添加一个新client实例后未更新配置，预期结果是沿用默认角色组配置，但系统会自动新增一个rolegroup1角色组。

2、修复集群机器/服务实例缩容后总览页面机器数量指标未进行相应更改。

3、修复多集群管理服务的keytab文件获取失败导致服务无法启动。

4、修复删除服务时同步删除对应告警。

5、修复多集群环境下cluster id传参不正确问题。

6、修复后台命令列表多页后半透明显示的bug。

7、修复多集群环境下全局变量获取失败的bug。

8、优化DataSophon Web服务启动脚本，移出部分内存参数。

9、修复偶发获取本地Actor失败的问题。

What's Changed

  • Dev 1.1.0 by @hyzhang13 in #158
  • [Feature] Check whether service on the same node. by @WujieRen in #161
  • perf: Remove constant Constants.FRAME_ID, and chagne the query method tp use SFunction. by @WujieRen in #162
  • A pr for issue #153: An odd number of zookpeer nodes must be deployed. by @WujieRen in #163
  • Variable declaration definition changed (var to let) by @zq0757 in #169
  • perf: Remove constant Constants.COMMAND_ID, and use SFunction instead. by @WujieRen in #171
  • datasophon step7.vue remove the extra letter y by @alldatadc in #172
  • perf: Remove constant Constants.SERVICE_ID and Constants.SERVICE_ROLE_TYPE. by @WujieRen in #174
  • [Feature] [.git] Added issue template by @zhu-mingye in #184
  • [Feature][*] Added Apache-License2 protocol header by @zhu-mingye in #179
  • [Feature] [.] Added spotless by @zhu-mingye in #188
  • [Feature-185][common] optimize read hosts file by @chenss-1 in #186
  • sql script character encoding is unified and standardized by @zq0757 in #189
  • fix(datasophon-common): 升级jsch版本 by @zhangdw123 in #182
  • [Improvement-190][MinaUtils] Improve get id_rsa file path to support different user home path on MinaUtils by @chenss-1 in #191
  • [Perf] Remove some constant in class Constants, and use SFunction instead. by @WujieRen in #200
  • A bug fix for querying the list of all service roles by @WujieRen in #202
  • [Bug] Solve the problem that when the namenode fails to format for the first time, subsequent commands cannot be executed by @WujieRen in #205
  • [Feature] Unified Front and back end by @liugddx in #192
  • [improvement] add dependency-review.yml and modify the build process by @gaotingkai in #199
  • fix(datasophon-api): 修复监控将fe配置成了be的问题以及格式化代码 by @zhangdw123 in #212
  • 调整JVM参数适应高版本 JDK，移除内存优化参数，部分JDK不适用，移除 DOLPHINSCHEDULER_LOG_DIR copy by @zhzhenqin in #213
  • [Improvement-214][Service packages] Upgrade Dinky to v0.7.2, DolphinScheduler to v3.1.4 by @leeoo in #215
  • 调整 ui 和 api 的打包结构，增加versions-maven-plugin by @zhzhenqin in #217
  • [#218][feat] Auto-start services when worker restart. by @WujieRen in #231
  • [Bug][Fix]Multiple cluster installation issues by @gmady520 in #244
  • Improvement-238 add dolphinscheduler script control_ds.sh by @chenss-1 in #245
  • 支持优先从worker templates下加载配置模版，其次从component根目录下加载 by @zhzhenqin in #240
  • fix the bug of adding starrocks backend and follower. by @zhangdw123 in #246

New Contributors

Full Changelog: v1.1.0...v1.1.1-beta

Contributors

leeoo, WujieRen, and 10 other contributors
Assets 3
Loading

v1.1.0 release

13 Feb 01:54
@datasophon datasophon
v1.1.0
c7e84c3
Compare
Choose a tag to compare
v1.1.0 release

v1.1.0

##New features
[New feature] Add integrated Kerberos security authentication.
[New features] New support for YARN, HDFS, ZK, Hive, Kafka, HBase component integration Kerberos, and friendly opening and closing of integrated Kerberos.
[New feature] New rack management function.
[New feature] The service installation command status is increased to cancel status.
[New feature] New tenant management function is added to facilitate the creation of operating system users and user groups.
[New feature] The specified user startup function is added. For example, Hdfs will use hdfs user startup management.
[New feature] When the new worker is stopped, the node exporter is also stopped.
##Bug fix
[Bug] Fix the problem that the database was not initialized during the Hive deployment installation

新特性

[新特性]新增集成Kerberos安全认证。

[新特性]新增支持YARN，HDFS，ZK，Hive，Kafka，HBase组件集成Kerberos，支持友好的开启和关闭集成Kerberos。

[新特性]新增机架管理功能。

[新特性]服务安装命令状态增加取消状态。

[新特性]新增租户管理功能，便于创建操作系统用户和用户组。

[新特性]新增指定用户启动功能，例如Hdfs将使用hdfs用户启动管理。

[新特性]新增Worker停止时，同时停止Node Exporter。

Bug修复

[Bug]修复Hive部署安装时，数据库没有初始化的问题

Assets 6
Loading

v1.0.2 release

09 Jan 12:41
@datasophon datasophon
v1.0.2
d9fbc22
Compare
Choose a tag to compare
v1.0.2 release

New Features

[New Feature] New service deletion function

[New feature] Use apache sshd to replace the function implemented by the original jsch to avoid the problem of jsch host link failure caused by the high version of openssh

[New feature] New service installation dependency verification function

[New feature] Kafka added the number of partitions below the minimum isr, offline directory, and the number of partitions in the unsynchronized state

[New features] kafka added overview chart topic write message trend chart per second, topic write data amount trend chart per second, kafka borker heap memory usage trend chart

[New feature] Add yarn used memory, yarn used cores index, and nodemanager lost index display

Enhance / Bug Fix

[Bug] Fix the Yarn memory usage. The Yarn kernel usage index shows abnormal problems

[Bug] Fix the problem of abnormal Yarn virtual core count and Yarn total virtual memory display

新特性

[新特性]新增服务删除功能

[新特性]使用apache sshd替换原jsch实现的功能，避免因openssh版本过高导致的jsch主机链接失败问题

[新特性]新增服务安装依赖校验功能

[新特性]kafka新增低于最小isr分区个数，离线目录，未同步状态分区数指标

[新特性]kafka新增总览图表topic每秒写入message趋势图，topic每秒写入数据量趋势图，kafka borker 堆内存使用率趋势图

[新特性]新增yarn已用内存，yarn已用核数指标，nodemanager lost个数指标展示

Bug修复

[Bug]修复yarn内存使用率，yarn内核使用率指标展示异常问题

[Bug]修复yarn虚拟核数，yarn总虚拟内存展示异常的问题

Assets 6
Loading

v1.0.1 release

28 Nov 14:30
@datasophon datasophon
v1.0.1
bf208d2
Compare
Choose a tag to compare
v1.0.1 release

New Features

[Feat] New added dolphinscheduler 3.1.1 support
[Feat] New added streampark 1.2.3 support

Enhance / Bug Fix

[Bug] Fix the problem of null pointexception when the session is null

[Bug] Fix the timeout failure of some service roles in Hdfs

[Bug] Fix the abnormal problem of obtaining host memory

[Bug] Fix the problem that elastic users do not have permission when deploying es

新特性

[新特性]新增集成dolphinscheduler 3.1.1
[新特性]新增集成streampark 1.2.3

Bug修复

[Bug] 修复session为null出现nullpointexception问题

[Bug] 修复Hdfs部分服务角色安装超时失败的问题

[Bug] 修复主机内存获取异常问题

[Bug] 修复es部署时，elastic用户没有权限的问题

Assets 4
Loading

v1.0.0

10 Nov 15:14
@datasophon datasophon
v1.0.0
92dd5ae
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.0.0 
Create LICENSE
Assets 4
Loading