Optional dependency for different storages?

[huxuan]

🚀 Feature Request

Although we support many kinds of storages, actually we only use one kind of them. So it would be great if we have different storages as different groups of optional dependencies. I have to admit this will bring back-compatibility issue. Just a head up here whether this could be possible to happen.

Motivation

We are migrating our code base from stratch and finding that streaming dataset introduces a lot of dependencies that we will not likely to use in the near future.

[Optional] Implementation

Additional context

[snarayan21]

Hey @huxuan, could you please clarify which dependencies you need and not need? Right now we have the large cloud providers as required dependencies (AWS, GCP, Azure, OCI) and other providers as optional (Databricks, Alipan, etc.)

[huxuan]

Hey @huxuan, could you please clarify which dependencies you need and not need? Right now we have the large cloud providers as required dependencies (AWS, GCP, Azure, OCI) and other providers as optional (Databricks, Alipan, etc.)

Thanks for the reply, actually we only use s3 compatible storage, so I suppose we only need boto3. A general idea is to have all the providers as optional. Maybe another approach is to provide a package only with the core functionalities and name it as something like streaming–core.