[Feature-3575][*] Force Task Success #4267

chengshiwen · 2020-12-20T12:31:35Z

Tips

Thanks very much for contributing to Apache DolphinScheduler.
Please review https://dolphinscheduler.apache.org/en-us/community/index.html before opening a pull request.

What is the purpose of the pull request

New Feature: Force Task Success #3726 #3575

Only add the ExecutionStatus FORCED_SUCCESS
Remove the operation ExecuteType resume_from_forced_success

this closes #3726, closes #3575, closes #196

Brief change log

Base on the previous pr for ui: [Feature][ui]Force success operation #3861 [Feature][ui] Resume from force success #4160, but there are some conflicts and the pr has not been updated for a long time
Base on the previous pr for api and master: [Feature-3575][*] Force Task Success #3576, but there are a lot of conflicts and the pr has not been updated for a long time
Merge the pr [Feature][ui]Force success operation #3861 [Feature][ui] Resume from force success #4160 [Feature-3575][*] Force Task Success #3576, and resolve all conflicts
Refactor some unreasonable code in frontend and backend
Remove the operation resume_from_forced_success
Fix the bug in frontend and backend joint debugging test
Fix some check style and unit test error
Remove redundant code

Verify this pull request

(Please pick either of the following options)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added dolphinscheduler-api tests in TaskInstanceControllerTest.java and TaskInstanceServiceTest.java for end-to-end.
Added dolphinscheduler-dao tests in TaskInstanceMapperTest.java for end-to-end.
Manually verified the change by testing locally for frontend to backend.

Gif Verify:

Video Verify: See Video Verify Link

update

更新本地的dev

1. 新添加executeType 2. 新添加对应的commandType 3. 修改controller对应的service部分 4. 补充processService中的verifyIsNeedCreateCommand的判断逻辑

1. 添加两个查询（mapper.java和mapper.xml） 2. 添加ProcessService中constructProcessInstance中对应的部分

update local dev

This reverts commit fa0bb5e.

…heduler into dev

1. 这样的话可以完整的构建整个dag，恢复之前的上下文；也没有任务可能重复执行的担忧 2. 不需要额外去处理process执行完之后的状态（主要是部分与整体的原因） 3. RecoverNodeIdList也不会重复

1. 失败任务强制成功后会继续submit后续结点 2. 失败重试的节点强制成功后会继续submit后面结点，并且停止retry 3. 对于失败的sub-process如果其中的结点强制成功了，在parent-process中不会有任何影响，只能等到结束后“从强制成功处继续执行”。

1. dataAnalysis 2. alertManager 3. businessTimeUtils的schedule选择

1. taskStateCount中添加了强制成功的状态 2. processStateCount中剔除强制成功的状态

regular update local dev

update local dev

防止logger注入的问题

codecov-io · 2020-12-20T15:35:20Z

Codecov Report

Merging #4267 (e3547a3) into dev (e2e6a77) will increase coverage by 0.02%.
The diff coverage is 55.00%.

@@             Coverage Diff              @@
##                dev    #4267      +/-   ##
============================================
+ Coverage     43.23%   43.26%   +0.02%     
- Complexity     3185     3193       +8     
============================================
  Files           471      471              
  Lines         21728    21796      +68     
  Branches       2626     2639      +13     
============================================
+ Hits           9395     9430      +35     
- Misses        11438    11469      +31     
- Partials        895      897       +2

Impacted Files	Coverage Δ	Complexity Δ
...heduler/api/controller/TaskInstanceController.java	`82.35% <0.00%> (-17.65%)`	`3.00 <0.00> (ø)`
...he/dolphinscheduler/common/utils/VarPoolUtils.java	`82.05% <0.00%> (ø)`	`7.00 <2.00> (ø)`
...heduler/server/master/runner/MasterExecThread.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...dolphinscheduler/common/enums/ExecutionStatus.java	`62.16% <66.66%> (+1.05%)`	`6.00 <0.00> (ø)`
...uler/api/service/impl/DataAnalysisServiceImpl.java	`98.91% <75.00%> (-1.09%)`	`30.00 <2.00> (+1.00)`	⬇️
.../apache/dolphinscheduler/api/dto/TaskCountDto.java	`79.16% <83.33%> (+1.38%)`	`7.00 <2.00> (+2.00)`
...apache/dolphinscheduler/api/enums/ExecuteType.java	`33.33% <100.00%> (ø)`	`1.00 <0.00> (ø)`
.../org/apache/dolphinscheduler/api/enums/Status.java	`100.00% <100.00%> (ø)`	`5.00 <0.00> (ø)`
...phinscheduler/api/service/TaskInstanceService.java	`98.38% <100.00%> (+0.71%)`	`15.00 <5.00> (+5.00)`
...inscheduler/service/zk/CuratorZookeeperClient.java	`60.97% <0.00%> (-4.88%)`	`7.00% <0.00%> (-1.00%)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2e6a77...e3547a3. Read the comment docs.

break60

front end +1

CalvinKirs

+1

lenboo · 2020-12-30T02:59:43Z

...-server/src/main/java/org/apache/dolphinscheduler/server/master/runner/MasterExecThread.java

@@ -414,7 +414,7 @@ private void initTaskQueue() {
            if (task.isConditionsTask() || DagHelper.haveConditionsAfterNode(task.getName(), dag)) {
                continue;
            }
-            if (task.getState().typeIsFailure() && !task.taskCanRetry()) {
+            if (task.getState().typeIsFailure() && !task.taskCanRetry() && !task.isConditionsTask()) {


already judged by the above if.

Ok, I will revise it

chengshiwen · 2020-12-30T03:49:07Z

I will remove the operation resume_from_forced_success and revise all code logic @lenboo @lgcareer @CalvinKirs @break60

davidzollo · 2021-01-05T14:20:06Z

I will remove the operation resume_from_forced_success and revise all code logic @lenboo @lgcareer @CalvinKirs @break60

any progress？

chengshiwen · 2021-01-06T02:57:32Z

I will remove the operation resume_from_forced_success and revise all code logic @lenboo @lgcareer @CalvinKirs @break60

any progress？

Not yet since preparing marriage propose in the past few days. I will finish it today or tomorrow.

chengshiwen · 2021-01-07T07:36:13Z

The operation resume_from_forced_success has been removed. Please review @dailidong @CalvinKirs @lenboo @lgcareer @break60

The configuration of force-task-B is as follow:

Gif Verify:

Video Verify: See Video Verify Link

sonarcloud · 2021-01-07T07:50:22Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

52.1% Coverage
0.0% Duplication

davidzollo

+1
great job, and the video gif is very clear

legen618 and others added 30 commits July 22, 2020 10:08

Merge pull request apache#1 from apache/dev

266f002

update

Merge pull request apache#2 from apache/dev

7f8d060

更新本地的dev

[implement] change task state

d86ceee

从强制成功的节点处启动-apiserver部分

3fceee9

1. 新添加executeType 2. 新添加对应的commandType 3. 修改controller对应的service部分 4. 补充processService中的verifyIsNeedCreateCommand的判断逻辑

从强制成功处启动-masterServer部分

5e9d664

1. 添加两个查询（mapper.java和mapper.xml） 2. 添加ProcessService中constructProcessInstance中对应的部分

fix bug in mapper

e912a80

[bugfix] issue apache#3324

fa0bb5e

Merge pull request apache#3 from apache/dev

d57daf0

update local dev

Revert "[bugfix] issue apache#3324"

23277bf

This reverts commit fa0bb5e.

Merge branch 'dev' of https://github.com/legen618/incubator-dolphinsc…

7cf55d5

…heduler into dev

支持sub-process和condition的TFS

9365376

调整数据库访问函数

5d02e30

改变原有的dag构建参数

18ae56b

1. 这样的话可以完整的构建整个dag，恢复之前的上下文；也没有任务可能重复执行的担忧 2. 不需要额外去处理process执行完之后的状态（主要是部分与整体的原因） 3. RecoverNodeIdList也不会重复

加入valid检验

d0e6b09

修改与新增commandType相关的代码

90a0601

1. dataAnalysis 2. alertManager 3. businessTimeUtils的schedule选择

添加dataAnalysis的状态统计相关的部分

c8fb897

1. taskStateCount中添加了强制成功的状态 2. processStateCount中剔除强制成功的状态

revert

079768f

增加了修改task状态的单元测试

815f62a

Merge pull request apache#4 from apache/dev

871be17

regular update local dev

add taskInstanceMapperTest

b418b68

update test of TaskInstanceController

1f19cf2

Merge pull request apache#5 from apache/dev

373cc47

update local dev

Merge branch 'dev' into dev-FTS

6bdd11d

fix

295796a

解决更新后的逻辑冲突

ae9eb16

fix import problems

1fc1218

Update DataAnalysisServiceImpl.java

6407745

adapt to code style

31cd1f6

fix vulnerability

d97cc3d

防止logger注入的问题

chengshiwen force-pushed the dev branch from 17ffaeb to fcb4e04 Compare December 20, 2020 15:26

fix code style and test

ae0ee3f

chengshiwen force-pushed the dev branch from fcb4e04 to ae0ee3f Compare December 20, 2020 15:41

break60 approved these changes Dec 22, 2020

View reviewed changes

chengshiwen force-pushed the dev branch from 092446c to 9e3aca2 Compare December 23, 2020 04:52

adjust locale for new pr

aa97b8a

chengshiwen force-pushed the dev branch from 9e3aca2 to aa97b8a Compare December 23, 2020 05:04

CalvinKirs approved these changes Dec 28, 2020

View reviewed changes

CalvinKirs requested review from lenboo and lgcareer December 28, 2020 06:47

lenboo reviewed Dec 30, 2020

View reviewed changes

Merge branch 'dev' into dev

325bc2e

davidzollo mentioned this pull request Jan 7, 2021

[Feature-3575][*] Force Task Success #3576

Closed

chengshiwen added 2 commits January 7, 2021 14:53

remove the operation resume_from_forced_success

5f39792

refactor code styles

e3547a3

chengshiwen force-pushed the dev branch from e4e417a to e3547a3 Compare January 7, 2021 07:32

davidzollo approved these changes Jan 10, 2021

View reviewed changes

davidzollo merged commit 21d1778 into apache:dev Jan 10, 2021

davidzollo mentioned this pull request Jan 10, 2021

[Feature][*] Force Task Success #3575

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature-3575][*] Force Task Success #4267

[Feature-3575][*] Force Task Success #4267

chengshiwen commented Dec 20, 2020 •

edited

Loading

codecov-io commented Dec 20, 2020 •

edited

Loading

break60 left a comment

CalvinKirs left a comment

lenboo Dec 30, 2020

chengshiwen Dec 30, 2020

chengshiwen commented Dec 30, 2020

davidzollo commented Jan 5, 2021

chengshiwen commented Jan 6, 2021

chengshiwen commented Jan 7, 2021 •

edited

Loading

sonarcloud bot commented Jan 7, 2021

davidzollo left a comment

[Feature-3575][*] Force Task Success #4267

[Feature-3575][*] Force Task Success #4267

Conversation

chengshiwen commented Dec 20, 2020 • edited Loading

Tips

What is the purpose of the pull request

Brief change log

Verify this pull request

codecov-io commented Dec 20, 2020 • edited Loading

Codecov Report

break60 left a comment

Choose a reason for hiding this comment

CalvinKirs left a comment

Choose a reason for hiding this comment

lenboo Dec 30, 2020

Choose a reason for hiding this comment

chengshiwen Dec 30, 2020

Choose a reason for hiding this comment

chengshiwen commented Dec 30, 2020

davidzollo commented Jan 5, 2021

chengshiwen commented Jan 6, 2021

chengshiwen commented Jan 7, 2021 • edited Loading

sonarcloud bot commented Jan 7, 2021

davidzollo left a comment

Choose a reason for hiding this comment

chengshiwen commented Dec 20, 2020 •

edited

Loading

codecov-io commented Dec 20, 2020 •

edited

Loading

chengshiwen commented Jan 7, 2021 •

edited

Loading