storage: add new backend HttpProxy
#974
Conversation
|
@loheagn , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/40296 |
|
@loheagn , The CI test is completed, please check result:
Congratulations, your test job passed! |
||||||||||||||||||||||||
|
Thanks @loheagn for the contribution, it can make My suggestion is that maybe we should implement the content store backend with HTTP over unix domain socket instead of TCP, which is simpler and requires only implementing two distribution APIs:
It's very like |
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/43987 |
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/43988 |
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/43992 |
api/src/config.rs
Outdated
| @@ -427,6 +441,14 @@ pub struct S3Config { | |||
| pub mirrors: Vec<MirrorConfig>, | |||
| } | |||
|
|
|||
| /// ContentProxyProxy configuration information to access blobs. | |||
| #[derive(Clone, Debug, Default, Deserialize, Eq, PartialEq, Serialize)] | |||
| pub struct LocalHttpProxyConfig { | |||
There was a problem hiding this comment.
How about LocalUdsConfig?
We can define an APIs to retrieve blobs from Unix domain socket, no matter it's containerd content store or something else.
There was a problem hiding this comment.
Or ContainerdContentStoreProxy? @imeoer
There was a problem hiding this comment.
But this backend driver only support http protocol now. I think LocalHttpProxy may be better.
There was a problem hiding this comment.
It's ok for me if using HttpProxy, maybe we can merge the feature with nydus-backend-proxy (not in this PR)? So that we can use HttpProxy to request an HTTP server that can proxy to local blob directory/containerd content store.
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/44026 |
|
@loheagn , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/44027 |
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/47782 |
|
@loheagn , The CI test is completed, please check result:
Sorry, your test job failed. Please get the details in the link. |
||||||||||||||||||||||||
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/47934 |
|
@loheagn , The CI test is completed, please check result:
Sorry, your test job failed. Please get the details in the link. |
||||||||||||||||||||||||
docs/nydusd.md
Outdated
| or | ||
|
|
||
| ``` | ||
| // for remote usage |
There was a problem hiding this comment.
Should be for local usage here.
This patch adds a new storage backend `HttpProxy` which can access blobs through a http proxy server. The http proxy server can be local (using unix socket) or remote (using `https://` or using `http://`). `HttpProxy` uses two API endpoints to access the blobs: - `HEAD /path/to/blobs` to get the blob size - `GET /path/to/blobs` to read the blob The http proxy server should respect [the `Range` header](https://www.rfc-editor.org/rfc/rfc9110.html#name-range) to compute the offset and length of the blob. The example config files for this new backend may be: ```jsonc // for remote usage { "backend": { "type": "http-proxy", "config": { "addr": "http://127.0.0.1:9977", "path": "/namespace/<repo>/blobs" } } } ``` or ```jsonc // for local unix socket { "backend": { "type": "http-proxy", "config": { "addr": "/path/to/unix/socket/file" } } } ``` There is also a test in `http_proxy.rs` to make sure `HttpProxy` works well, which setups a simple http server and generates a `HttpProxy` backend to get contents from the server. Signed-off-by: Nan Li <loheagn@icloud.com>
|
@loheagn , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/48133 |
|
@loheagn , The CI test is completed, please check result:
Sorry, your test job failed. Please get the details in the link. |
||||||||||||||||||||||||
|
need rebase to master since anolis test script is updated |
This pr adds a new storage backend
HttpProxywhich can access blobs through a http proxy server.The http proxy server can be local (using unix socket) or remote (using
https://or usinghttp://).HttpProxyuses two API endpoints to access the blobs:HEAD /path/to/blobsto get the blob sizeGET /path/to/blobsto read the blobThe http proxy server should respect the
Rangeheader to compute the offset and length of the blob.The example config files for this new backend may be:
or
There is also a test in
http_proxy.rsto make sureHttpProxyworks well, which setups a simple http server and generates aHttpProxybackend to get contents from the server.Signed-off-by: Nan Li loheagn@icloud.com