[DSIP-19][Feature] Add connection center feature for DS

[EricGao888]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

• Add Connection Center feature for Dolphin Scheduler so that plugins will be able to achieve connection information such as host, endpoint, credentials to external system and switch them easily without restarting DS.

Use case

• For more information of this feature, we could refer to Apache Airflow Connections Management

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

[EricGao888]

Could anyone please help add the discussion label to this issue? I think we need some discussions for the design of this feature : )

[davidzollo]

good idea

[EricGao888]

This feature would be something like extended datasource feature which we already have. The difference is that all task plugins besides SQL related ones could gain access to it.

[EricGao888]

BTW, as discussed, we need some kind of data masking mechanism to hide sensitive data. Should we open another issue for data masking?

[EricGao888]

BTW, as discussed, we need some kind of data masking mechanism to hide sensitive data. Should we open another issue for data masking?

Added here: #10498

[SbloodyS]

It seems that this is more like a configuration center in ds. WDYT? @EricGao888

[EricGao888]

It seems that this is more like a configuration center in ds. WDYT? @EricGao888

@SbloodyS Exactly.

[EricGao888]

Seems there would be two kinds of configurations if we implement this feature on the basis of current data center.

1. Connection related configurations for datasource, task plugins, etc.
2. Some common configurations which are in common.properties at present.

BTW, we may still keep less-frequently-changed and high-risky configurations in common.properties.

[EricGao888]

Could you please help add a discussion label for this issue? Thx @SbloodyS

[SbloodyS]

Could you please help add a discussion label for this issue? Thx @SbloodyS

Sure.

[SbloodyS]

Seems there would be two kinds of configurations if we implement this feature on the basis of current data center.

1. Connection related configurations for datasource, task plugins, etc.
2. Some common configurations which are in common.properties at present.

BTW, we may still keep less-frequently-changed and high-risky configurations in common.properties.

Yes. I think the data source center should be part of the configuration center.

[EricGao888]

Some ideas about user experience:

• We will have each task plugin bound with default connection, which corresponds to task types.
• We set default values in default connections so that new users to Dolphin Scheduler will not need to spend time figuring out what this configuration center does if they just try some basic stuff.
• Users will be able to see default example values in connection related configurations to help them understand how to manage connections.

[EricGao888]

Rules for configuration update should be established at the very beginning to avoid confusion.

[EricGao888]

I will give a design later this month. For the first stage, we will focus on task plugin connections part.

[qingwli]

Hi @EricGao888 , Could you help realize multiple resource centers?

[EricGao888]

Hi @EricGao888 , Could you help realize multiple resource centers?

We have already removed singleton design for storage operators in backend. Therefore, it is possible to support multiple remote resource storages in resource center. But we need a good design for UI interactions.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.