Research OpenDAL for possible usages

[spencergilbert]

https://github.com/datafuselabs/opendal

Likely requires an RFC prior to implementation, but a spike could be interesting.

OpenDAL today appears to support the following services that Vector has components for today:

• AWS S3
• Azure Storage Blob
• FS(?)
• Google Cloud Storage
• HDFS
• Redis

As well as HDFS (which has been requested in the past), among a number of others. It's possible we could replace the disparate dependencies of these components and just use OpenDAL as a shared interface.

sccache recently started using OpenDAL for it's remote storage needs.

[Xuanwo]

Hi, OpenDAL's maintainer is here! ❤️

Thanks for the researching, and I'm willing to answer any potential questions if needed.

[Xuanwo]

Based on my experience with sccache's OpenDAL migration, we can evaluate OpenDAL by adding new service support. For example, we can use OpenDAL to implement hdfs/oss/... support (I'm interested in helping with this!).

This way:

• We don't break existing features
• OpenDAL can implement features that are needed by Vector.
• We can release new features!

After the feature has been successfully released, we can try to migrate existing services one by one. Since the API provided by OpenDAL is elegant (yes, I'm proud of that!), the migration should be as easy as removing existing code.

[spencergilbert]

@Xuanwo sink side is more obvious to me on how OpenDAL could be integrated. Do you have any initial thoughts on how a source could leverage it? Today we're relying on notifications from SQS to know when to read from S3.

At a glance nothing like that's included in OpenDAL - so we'd still need to catch those and then use OpenDAL to pull the related blobs?

[Xuanwo]

At a glance nothing like that's included in OpenDAL - so we'd still need to catch those and then use OpenDAL to pull the related blobs?

I'm afraid so. OpenDAL can't handle notifications services like SQS.

[spencergilbert]

I think a good target to spike would be a hdfs sink, is that something you're interested in trying and contributing @Xuanwo or should I add this to our backlog to test when we get the time?

[Xuanwo]

I think a good target to spike would be a hdfs sink, is that something you're interested in trying and contributing @Xuanwo or should I add this to our backlog to test when we get the time?

Cool, let's do this!

[spencergilbert]

We're trying to wrap up some guidance for contributing new sinks that should hopefully be useful to reference - #16070

[Xuanwo]

We're trying to wrap up some guidance for contributing new sinks that should hopefully be useful to reference - #16070

Some progress report: I'm working on this now. I expect to submit a draft for review later this week!

[Xuanwo]

PR landed in #16399. Maybe we can add hdfs first and than adding more features?

[Xuanwo]

The pull request #16557 has been merged for some time now. I am considering writing documentation or a post on how to implement a sink using OpenDAL, in order to encourage others to participate in the development process. Which approach do you prefer?

[spencergilbert]

Hey @Xuanwo I'll discuss with @jszwedko this week, I think we'll want to do a check to make sure we've got feature parity between OpenDAL based sinks and our existing sinks - but I'll let you know ASAP.

Congrats on the Apache incubation!

[gaby]

Related to #3382