Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Look for rusoto alternatives #131

Closed
aaronmondal opened this issue Jul 4, 2023 · 5 comments · Fixed by #369
Closed

Look for rusoto alternatives #131

aaronmondal opened this issue Jul 4, 2023 · 5 comments · Fixed by #369
Assignees
Labels
no-effect-in-production security Issues that have security implications

Comments

@aaronmondal
Copy link
Member

Unfortunately rusoto is in "maintenance mode".

Fortunately, this only seems to affect cas/store/s3_store.rs and its tests.

It seems that https://github.com/awslabs/aws-sdk-rust could be a good alternative. That's also used by attic, which is basically turbo-cache for nix: https://github.com/zhaofengli/attic/blob/main/server/src/storage/s3.rs

@allada
Copy link
Member

allada commented Jul 5, 2023

Yeah, I've been wanting to get off of rusoto for a while. I don't believe aws-sdk-rust existed when I first needed an s3 client. but yeah aws-sdk-rust is a good alternative, other people have suggested it to me.

@aaronmondal
Copy link
Member Author

@allada If you're ok with it I'd like to work on this. My plan would be to replicate the existing functionality and tests and then attempt adding some integration tests that interact with a "real" bucket. I'll probably need to get turbo-cache running in k8s first, but a naive, non-cloud-native way seems fairly straightforward now that I got #178 working ☺️

Regarding integration tests, maybe we can get away with using something like LocalStack in CI?

@allada
Copy link
Member

allada commented Jul 14, 2023

Yeah I'm super happy to have you take this.

For testing, I'd rather try to keep it all to unit tests until we have more users. Try not to focus much on getting tests to work with it, once we get a basic example working, I can go over it and see what the best strategy to test it is. If we need we can write a wrapper around the API calls we need to make and then mock out the implementation... but I'd like to avoid that if possible.

@aaronmondal aaronmondal self-assigned this Jul 15, 2023
@chrisstaite-menlo
Copy link
Collaborator

We have a dependabot notification that "time" is vulnerable to a Segmentation Fault. This crate is old required by "chrono" which is only required by "rusoto", since this is not maintained we have no way to fix this issue other than to move to an alternative crate for rusoto.

@chrisstaite-menlo chrisstaite-menlo added the security Issues that have security implications label Sep 13, 2023
@allada
Copy link
Member

allada commented Sep 13, 2023

Yeah this is on my todo list.

aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 1, 2023
This change removes the rusoto dependency and substitutes it with the
AWS SDK for Rust.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 1, 2023
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 1, 2023
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 6, 2023
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 6, 2023
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 6, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- We now support Http2 via hyper/rustls.
- The previous concurrency limit is now a limit per store, as the
  limiting factor is S3 blocking due to excessive connection creation.
  The new per-store limit is now 4.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 6, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 7, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 7, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 7, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 11, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.
- Multipart uploads now run out of order.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 11, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.
- Multipart uploads now run out of order.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 11, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.
- Multipart uploads now run out of order.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 11, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.
- Multipart uploads now run out of order.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 13, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.
- Multipart uploads now run out of order.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 16, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now uses HTTP/2 via hyper/rustls.
- The concurrency limits have been removed. Each store now uses a single
  conection with request multiplexing being handled via HTTP/2.
- Multipart uploads now run out of order.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 18, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 18, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 19, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 19, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 19, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 19, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit to aaronmondal/nativelink that referenced this issue Nov 20, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
aaronmondal added a commit that referenced this issue Nov 20, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes #131
TripleKai pushed a commit to TripleKai/turbo-cache that referenced this issue Nov 29, 2023
This is a breaking change.

- The S3 store now uses the aws-sdk for Rust to handle higher level
  interactions with the S3 API.
- It now also supports HTTP/2 via hyper/rustls.
- The concurrency limits have been removed and are now only configurable
  for multipart uploads.
- The default behavior is now HTTPS. To enable unencrypted
  communication, an `insecure_allow_http` flag has been added.

Fixes TraceMachina#131
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no-effect-in-production security Issues that have security implications
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants