Feature request: support passing customized connector or connector adapter

[wenym1]

Motivation

We are using AWS S3 or any S3 compatible object store as our storage backend. Previously we use aws-sdk-s3 to access the object store, and recently we are migrating from the aws sdk to opendal.

Previously when we use the aws sdk, it supports passing customized http client (see method http_client). In our code, we build our own http client to apply some customized logic. Most logics can be supported in opendal, such as setting nodelay, keepalive. The only thing left is passing a customized connector. When building a hyper client by the build method, it support passing a customized connector that implements the Connect trait. The trait is accessed when creating a new connection. We leverage this feature to monitor the tcp connection, such as number of existing connection, throughput of each connection etc. We implemented our own connector that wraps the original hyper http connector and impl the Connect trait in risingwavelabs/risingwave#11848. However, in opendal this feature is not supported yet.

Requested feature

I think there are two ways to support the feature.

The first one is to support passing a customized http client or a connector like what the aws sdk does. This may leak too much flexibility to our external users, as it depends on users to properly initialize the connector.

The second one is to support passing a connector adapter with the following definition

trait ConnectorAdapter {
    fn wrap(connector: impl Connect) -> impl Connect;
}

With this adapter trait, the opendal can initial the connector internally, and then wrap the initialized connector with this adapter, and in the end pass it to the http client.

Implementation-wise consideration

It seems that opendal is using reqwest to make the http request. I took a rough look at the code of reqwest. Though its underlying http client is also hyper, in here, it seems that the connector used by the http client is fixed. So to support this feature, we shall either modify the reqwest code to support passing a customized connector, or change to make the http request from the raw hyper client.

[Xuanwo]

Thanks for raising the issue!

I'm open to switching our HTTP client from reqwest to hyper, which offers us greater control over the underlying HTTP operations.

I can see the following benefits:

• Upgrade to hyper 1 for now (good to our users dependences)
• More control on http (like in mentioned by OP)
• Async Runtime Agnostic

Based on our current codebase, here are several notes:

Good

OpenDAL wraps all http client side code in http_util, we don't need to replace every usage of http client.

Bad

OpenDAL now relies on the following features provided by reqwest:

• system proxy (should be simple I guess)
• HTTP Upgrade (from http/1.1 to http/2)
• HTTP 1.1 Keep Alive (a.k.a connection pool)

[Xuanwo]

I think about this again and plan to introduce a HttpClient trait:

trait HttpClient {
  async fn send(&self, req: Request<Buffer>) -> Response<Buffer>;
}

You can do whatever you want to the client itself (you can even use the same client from aws sdk!).

What do you think? @wenym1

[wenym1]

I think about this again and plan to introduce a HttpClient trait:

trait HttpClient {
  async fn send(&self, req: Request<Buffer>) -> Response<Buffer>;
}

You can do whatever you want to the client itself (you can even use the same client from aws sdk!).

What do you think? @wenym1

Looks promising to me!

If you have any PR on it, could you invite me to review so that I can try whether the new trait can satisfy our usage? Thanks!

[Xuanwo]

Hi @wenym1, this feature has been implemented as seen in #4471. We plan to include it in v0.50.1, our upcoming release.