[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

xdu-chenrj · 2023-06-30T16:14:17Z

Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks

issue: #14338

Related issues: [DSIP-19][Feature] Add connection center feature for DS #10283

Mail: https://lists.apache.org/thread/xl6pb3sbrt0ffrf1fltcph39s9w1pjlx

…l connections in zeppelin tasks

xdu-chenrj · 2023-06-30T16:17:25Z

Tested locally.

xdu-chenrj · 2023-06-30T16:32:23Z

@EricGao888
I want to obtain the type parameter of the zeppelin task, but I am not sure how it is transmitted from the front-end. In order to quickly advance, I directly wrote a fixed value.

When building connection parameters in DataSourceUtils, it will be built based on the type of task.

 DataSourceUtils.buildConnectionParams(DbType.valueOf("ZEPPELIN"), zeppelinTaskExecutionContext.getConnectionParams());

Radeity · 2023-07-01T02:22:38Z

Hi, @xdu-chenrj , thanks for your contribution :D

Some suggestions for you before taking further look of this PR :

You should better follow the PR template to describe this PR.
Please add missing license header for dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-zeppelin/pom.xml.
Please avoid using wild card import in codes, e.g. import org.apache.dolphinscheduler.plugin.task.api.*;.
Please run mvn spotless:apply to format your code first.
Please handle conflicts first.

xdu-chenrj · 2023-07-01T04:36:14Z

Hi, @xdu-chenrj , thanks for your contribution :D

Some suggestions for you before taking further look of this PR :

You should better follow the PR template to describe this PR.

Please add missing license header for dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-zeppelin/pom.xml.

Please avoid using wild card import in codes, e.g. import org.apache.dolphinscheduler.plugin.task.api.*;.

Please run mvn spotless:apply to format your code first.

Please handle conflicts first.

Okay, I got it

…e code

.asf.yaml

deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl

...er-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/DataSourceServiceImpl.java

EricGao888

BTW, could u plz update related UT cases for Zeppelin tasks?

xdu-chenrj · 2023-07-08T03:17:57Z

BTW, could u plz update related UT cases for Zeppelin tasks?

Okay, I got it

codecov-commenter · 2023-07-21T12:54:47Z

Codecov Report

Merging #14434 (28aaf14) into dev (c0ed681) will increase coverage by 0.05%.
The diff coverage is 59.79%.

❗ Current head 28aaf14 differs from pull request most recent head 2a3b90f. Consider uploading reports for the commit 2a3b90f to get more accurate results

@@             Coverage Diff              @@
##                dev   #14434      +/-   ##
============================================
+ Coverage     38.09%   38.15%   +0.05%     
- Complexity     4642     4660      +18     
============================================
  Files          1262     1269       +7     
  Lines         45155    45238      +83     
  Branches       4944     4947       +3     
============================================
+ Hits          17203    17259      +56     
- Misses        26060    26087      +27     
  Partials       1892     1892

Files	Coverage Δ
...rce/zeppelin/param/ZeppelinDataSourceParamDTO.java	`0.00% <0.00%> (ø)`
...uler/plugin/task/zeppelin/ZeppelinTaskChannel.java	`0.00% <0.00%> (ø)`
...in/task/zeppelin/ZeppelinTaskExecutionContext.java	`80.00% <80.00%> (ø)`
.../org/apache/dolphinscheduler/spi/enums/DbType.java	`0.00% <0.00%> (ø)`
...datasource/zeppelin/ZeppelinDataSourceChannel.java	`0.00% <0.00%> (ø)`
...rce/zeppelin/ZeppelinDataSourceChannelFactory.java	`0.00% <0.00%> (ø)`
...gin/datasource/zeppelin/ZeppelinClientWrapper.java	`71.42% <71.42%> (ø)`
...uler/plugin/datasource/zeppelin/ZeppelinUtils.java	`0.00% <0.00%> (ø)`
...duler/plugin/task/zeppelin/ZeppelinParameters.java	`58.33% <60.00%> (+25.00%)`	⬆️
...inscheduler/plugin/task/zeppelin/ZeppelinTask.java	`55.33% <62.50%> (+5.33%)`	⬆️
... and 1 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

sonarcloud · 2023-07-21T13:05:32Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

EricGao888 · 2023-10-30T06:48:18Z

@xdu-chenrj Could please resolve the conflicts? Together with other PRs related to DSIP-19. Thanks : )

EricGao888

Waiting for CI to pass

sonarcloud · 2023-10-30T08:42:55Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

EricGao888 · 2023-10-30T08:45:29Z

Thanks for helping improve zeppelin task plugin @xdu-chenrj

...va/org/apache/dolphinscheduler/plugin/datasource/zeppelin/param/ZeppelinConnectionParam.java

+
+import com.fasterxml.jackson.annotation.JsonInclude;
+
+@Data


...va/org/apache/dolphinscheduler/plugin/datasource/zeppelin/param/ZeppelinConnectionParam.java

+
+    protected String username;
+
+    protected String password;


…ell as external connections to the connection center in zeppelin tasks (apache#14434) * Refactoring zeppelin task plugin with connections managed in connection center --------- Co-authored-by: Eric Gao <ericgao.apache@gmail.com>

support for zeppelin connections in the connection center and externa…

d596593

…l connections in zeppelin tasks

xdu-chenrj requested review from kezhenxu94, songjianet, Amy0104, caishunfeng, SbloodyS and zhuangchong as code owners June 30, 2023 16:14

github-actions bot added UI ui and front end related backend e2e e2e test labels Jun 30, 2023

Radeity assigned xdu-chenrj Jul 1, 2023

xdu-chenrj added 2 commits July 1, 2023 14:14

synchronized the code and resolved conflicts, as well as formatted th…

ff051a2

…e code

repaired the method for retrieving task types

ab40648

EricGao888 added improvement make more easy to user or prompt friendly 3.2.0 for 3.2.0 version labels Jul 6, 2023

EricGao888 added this to the 3.2.0 milestone Jul 6, 2023

EricGao888 reviewed Jul 6, 2023

View reviewed changes

.asf.yaml Outdated Show resolved Hide resolved

EricGao888 reviewed Jul 6, 2023

View reviewed changes

deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl Outdated Show resolved Hide resolved

EricGao888 reviewed Jul 6, 2023

View reviewed changes

...er-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/DataSourceServiceImpl.java Outdated Show resolved Hide resolved

EricGao888 reviewed Jul 6, 2023

View reviewed changes

songjianet previously approved these changes Jul 7, 2023

View reviewed changes

kezhenxu94 removed their request for review July 7, 2023 01:23

recovered code with incorrect changes

c0bf072

xdu-chenrj dismissed songjianet’s stale review via c0bf072 July 8, 2023 03:41

xdu-chenrj and others added 10 commits July 17, 2023 15:02

Merge branch 'dev' into dev

71d8ec6

Merge branch 'dev' into dev

3976a0f

Merge branch 'dev' into dev

5f1e3bc

Merge branch 'dev' into dev

444265f

fixed zeppelin task UT

9f4a0f0

fixed zeppelin task UT

50fb4cb

resolved the conflict

936a98c

modified zeppelin code

a335054

Merge branch 'dev' into dev

ec92af1

Merge branch 'dev' into dev

036fda4

resolved the conflict

62aa088

zhongjiajie modified the milestones: 3.2.0, 3.3.0 Aug 30, 2023

xdu-chenrj added 4 commits September 14, 2023 15:01

resolved the conflict

779af0c

supported zeppelin connection

b8165a4

Merge remote-tracking branch 'upstream/dev' into dev

b70d342

Merge remote-tracking branch 'upstream/dev' into dev

e43e8b4

resolved the conflict

2a3b90f

EricGao888 added the ready-to-merge label Oct 30, 2023

EricGao888 approved these changes Oct 30, 2023

View reviewed changes

EricGao888 merged commit e99c595 into apache:dev Oct 30, 2023
53 of 55 checks passed

github-advanced-security bot found potential problems Oct 30, 2023

View reviewed changes

SbloodyS added DSIP and removed 3.2.0 for 3.2.0 version labels Aug 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

xdu-chenrj commented Jun 30, 2023

xdu-chenrj commented Jun 30, 2023

xdu-chenrj commented Jun 30, 2023

Radeity commented Jul 1, 2023 •

edited

Loading

xdu-chenrj commented Jul 1, 2023

EricGao888 left a comment

xdu-chenrj commented Jul 8, 2023

codecov-commenter commented Jul 21, 2023 •

edited

Loading

sonarcloud bot commented Jul 21, 2023

EricGao888 commented Oct 30, 2023

EricGao888 left a comment

sonarcloud bot commented Oct 30, 2023

EricGao888 commented Oct 30, 2023

[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

[DSIP-19] Support zeppelin connections in the connection center, as well as external connections to the connection center in zeppelin tasks #14434

Conversation

xdu-chenrj commented Jun 30, 2023

xdu-chenrj commented Jun 30, 2023

xdu-chenrj commented Jun 30, 2023

Radeity commented Jul 1, 2023 • edited Loading

xdu-chenrj commented Jul 1, 2023

EricGao888 left a comment

Choose a reason for hiding this comment

xdu-chenrj commented Jul 8, 2023

codecov-commenter commented Jul 21, 2023 • edited Loading

Codecov Report

sonarcloud bot commented Jul 21, 2023

EricGao888 commented Oct 30, 2023

EricGao888 left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Oct 30, 2023

EricGao888 commented Oct 30, 2023

Radeity commented Jul 1, 2023 •

edited

Loading

codecov-commenter commented Jul 21, 2023 •

edited

Loading