Can I ask you the difference between this and rust-s3?

[frederikhors]

Can I ask you the difference between this and rust-s3?

I'm trying for the first time rust-s3 today and for the first time Rust in general and I have this issue:

tokio-rs/axum#1638.

Can I ask for your help? And maybe how to use rusty-s3 to fix it?

Thanks! 😃

[paolobarbolini]

What you're trying to do should be fairly easy to achieve with rusty-s3, reqwest, headers and axum. Here's a rough guide:

Uploading

1. Use a BodyStream extractor to obtain the http request body as a Stream of Bytes. You can also use Multipart but it requires a few extra steps
2. Use the TypedHeader<headers::ContentLength> extractor to obtain the request body Content-Length
3. Do a put_object call using rusty-s3, similar to the last 3 lines of this example: https://docs.rs/rusty-s3/0.3.2/rusty_s3/index.html#basic-getting-started-example
4. Use the reqwest crate to make a PUT request using the signed URL generated by rusty-s3, adding a Content-Length header with the value of the previously obtained Content-Length and the BodyStream as the body of the request

Downloading

1. Do a get_object call using rusty-s3, similar to the last 3 lines of this example: https://docs.rs/rusty-s3/0.3.2/rusty_s3/index.html#basic-getting-started-example
2. Use the reqwest crate to make a GET request using the signed URL generated by rusty-s3
3. Optionally get the Content-Length of the s3 response and apply it to the axum response
4. Optionally set a Content-Type and a Content-Disposition header
5. Use the reqwest Bytes stream as the axum response body

[frederikhors]

Thank you. I'll try. What do you think about creating an example with the very popular router axum?

This is a very needed feature for almost any middle-big app.

[frederikhors]

@paolobarbolini, I'm following your suggestion. The download part works good! Thanks!

For the upload I'm using the below code and it works until I need to post() with reqwest:

#[async_trait::async_trait]
impl Trait for UploadClient {
    async fn put_file(
        &self,
        filename: &str,
        mut reader: Pin<&mut (dyn AsyncRead + Send + Sync)>,
    ) -> Result<u64> {
        // let bucket = ...;
        // let credentials = ...;

        let presigned_url_duration = Duration::from_secs(60 * 60);

        let action = bucket.put_object(Some(&credentials), filename);

        let client = reqwest::Client::new();

        let resp = client
            .post(action.sign(presigned_url_duration))
            // Nothing of these works!
            // .body(reader)
            // .body(reqwest::Body::wrap_stream(reader))
            // .body(ReaderStream::new(reader))
            .body(StreamReader::new(reader))
            .send()
            .await?
            .error_for_status()?;

        Ok(123)
    }
}

What can I use for reqwest to post the AsyncRead stream?

[paolobarbolini]

You'll need to convert the AsyncRead into a Stream of Bytes using this: https://docs.rs/tokio-util/0.7.4/tokio_util/io/struct.ReaderStream.html

[frederikhors]

It works. Now the last problem is: https://stackoverflow.com/questions/74812937/why-the-error-cannot-borrow-multipart-as-mutable-more-than-once-at-a-time. Can you help me fix it?

[frederikhors]

To give you more context that you rightfully need I've created a small Github repo that mimics exactly what I'm trying to achieve: streaming file uploads to both file systems and the cloud (S3) using axum (and reqwest).

I can change all the code you see. There is no restriction. All signatures were invented by me who are in my early days with the awesome Rust.

If you want I can also eliminate reqwest, as long as I understand what to use instead! 😄