[DSIP-17][Feature][Registry] Support etcd as registry

[ruanwenjun]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Right now, we use zookeeper as registry and store metadata of master and worker.

We have already implemented the registry plug-in architecture, it's time to support etcd as a new registry plugin choose. This can help user who only familiar with etcd to use dolphinscheduler.

I want to make this feature as a task in GSOC 2022. @caishunfeng

Use case

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[EricJoy2048]

That's great!

[txfs19260817]

Hello everyone, I would like to contribute to this project for GSoC 2022. Is this still open?

About me: my name is Zishuo Ding and I am currently a graduate student at CMU. I have some experience in both working and contributing to open source.

Thank you!

[ruanwenjun]

Hello everyone, I would like to contribute to this project for GSoC 2022. Is this still open?

About me: my name is Zishuo Ding and I am currently a graduate student at CMU. I have some experience in both working and contributing to open source.

Thank you!

Hi Zishuo, this project is still open now, you can follow the GSOC process to apply.

[ruanwenjun]

@txfs19260817 Hi, zishuo, it's time to submit a proposal on gsoc site, don't forget.

[txfs19260817]

Dear mentor @ruanwenjun , apologies for the delayed response, because I am now overburdened with exams and assignments. I've started a discussion on the slack channel #gsoc. I'm hoping we can exchange some information on this project, proposal and other things linked to it. Thanks!

[caishunfeng]

Hi @txfs19260817 welcome to join this feature.

[remjacob]

It's great. We are waiting this feature!

[wjf222]

Design

CRUD

Etcd is a distributed key-value storage system which support CRUD originally.
For some temp keys which need to be deleted on discoonect. we can grant leases for keys from etcd cluster.When a key is attached to a lease, its lifetime is bound to the lease's lifetime which in turn is governed by a time-to-live(TTL).Once a lease's TTL elapses, the lease expires and all attached keys are deleted.Applications can keep a lease alive by refreshing its TTL so it does not expire. By lease and keepalive, we can make sure the key will exist when the client disconnect, the lease will delete.

Subscribe/Notify

There is a watchClient in etcd to monitor changes to keys. When a watch response coming,the listener will notify.

Lock

Etcd Lock uses a lease on a key which is revoked when the lock is reased.If the server the lock is running on dies, or the network is disconnected, etcd will time out the lock.

ConnectionStateListener

We get Client's shared channel by reflect, becase all fields is private on jetcd.
There will be a scheduler thread, if the channel state has changed, then is will trigger the connection listener.

[caishunfeng]

@ruanwenjun maybe we should take a DSIP of registry, and this issue as the sub issue of it, WDYT?

[ruanwenjun]

@ruanwenjun maybe we should take a DSIP of registry, and this issue as the sub issue of it, WDYT?

Agree with you, but I just add DSIP label for this issue, I think we need to create a ISSUE to collect all DSIP.