Support Multipart Forms

[SergioBenitez]

Rocket currently doesn't provide built-in support for multipart forms, but it should!

[flosse]

Do you know a quick workaround to support it?

[flosse]

I tried to use mime_multipart::read_multipart like this:
read_multipart(&mut data.open(),true)
but if fails :(

[SergioBenitez]

You should be able to adapt the multipart crate pretty easily here. I can't vouch for it, however, as I haven't tried it myself.

[flosse]

Thanks for that fast reply!

[SergioBenitez]

Sure! Grabbing lunch now, but can whip something up afterward!

[Zeludon]

How was the lunch, more importantly how is multipart integration coming?

[SergioBenitez]

@Zeludon Lunch was great, thanks! 😋

The simplest route towards multipart support is via an existing crate. The most complete of those available is multipart. Unfortunately its API is rather restrictive at the moment, and so it wouldn't be a good fit for Rocket. Nonetheless, I've proposed a change to the API to the author/maintainer at abonander/multipart#58 that would allow its integration with Rocket.

[Maaarcocr]

I'm working on a small utility website where I need to upload a csv file from a form. Has there been any progress for this?

Thanks for this great library, I'm in love with it!

[abonander]

@SergioBenitez If there's any more blockers, let me know.

[shepmaster]

I answered a related question on Stack Overflow. Note that there are limitations of this approach.

#![feature(plugin)]
#![plugin(rocket_codegen)]

extern crate rocket;
extern crate multipart;

#[post("/", data = "<upload>")]
fn index(upload: DummyMultipart) -> String {
    format!("I read this: {:?}", upload)
}

#[derive(Debug)]
struct DummyMultipart {
    alpha: String,
    one: i32,
    file: Vec<u8>,
}

use std::io::{Cursor, Read};
use rocket::{Request, Data, Outcome};
use rocket::data::{self, FromData};
use multipart::server::Multipart;

impl FromData for DummyMultipart {
    type Error = ();

    fn from_data(request: &Request, data: Data) -> data::Outcome<Self, Self::Error> {
        // All of these errors should be reported
        let ct = request.headers().get_one("Content-Type").expect("no content-type");
        let idx = ct.find("boundary=").expect("no boundary");
        let boundary = &ct[(idx + "boundary=".len())..];

        let mut d = Vec::new();
        data.stream_to(&mut d).expect("Unable to read");

        let mut mp = Multipart::with_body(Cursor::new(d), boundary);

        // Custom implementation parts

        let mut alpha = None;
        let mut one = None;
        let mut file = None;

        mp.foreach_entry(|mut entry| {
            match entry.name.as_str() {
                "alpha" => {
                    let t = entry.data.as_text().expect("not text");
                    alpha = Some(t.into());
                },
                "one" => {
                    let t = entry.data.as_text().expect("not text");
                    let n = t.parse().expect("not number");
                    one = Some(n);
                },
                "file" => {
                    let mut d = Vec::new();
                    let f = entry.data.as_file().expect("not file");
                    f.read_to_end(&mut d).expect("cant read");
                    file = Some(d);
                },
                other => panic!("No known key {}", other),
            }
        }).expect("Unable to iterate");

        let v = DummyMultipart {
            alpha: alpha.expect("alpha not set"),
            one: one.expect("one not set"),
            file: file.expect("file not set"),
        };

        // End custom

        Outcome::Success(v)
    }
}

fn main() {
    rocket::ignite().mount("/", routes![index]).launch();
}

$ curl -X POST -F alpha=omega -F one=2 -F file=@hello http://localhost:8000/
I read this: DummyMultipart { alpha: "omega", one: 2, file: [104, 101, 108, 108, 111, 44, 32, 119, 111, 114, 108, 100, 33, 10] }

[SergioBenitez]

@shepmaster Thanks for the snippet! One particular concern with your code is that you read all of the form data into memory with no limit. This means that, for instance, if a user uploads a 500MB file, the entire file will reside in memory even though it could have been streamed to disk. Or, worse, if a user sends you data infinitely, the request will exhaust the memory resources on the server, likely crashing it.

[abonander]

What's the status on native integration? Not much else I can do on my end, AFAIK.

[shepmaster]

@SergioBenitez absolutely, and thank you for pointing that out. However, many people don't need to worry about that level security for their own application for whatever reason. A solution in the hand is worth two in the bush. 😺

[SergioBenitez]

@abonander My primary concern is correctness. Given that each time (three so far) I've tried to use your library I've run into showstoppers regarding correctness, I'm not particularly confident that it will function as required. I'd like to see abonander/multipart#59 closed. Parsing and networking bugs can be some of the most insidious, and they can lead to extreme confusion when they pop up. I'd really, really like to avoid this.

[abonander]

I tried fuzzing and it found several more bugs; last I tried it, I believe it ran for several hours without issue. Should I let it run overnight? For a week? A month? Should I have it run on CI somehow? I haven't seen any good rule of thumb here.

I've added a number of edge cases to the integration test. No new issues have cropped up, unless there's some you haven't told me about.

[SergioBenitez]

@abonander Are you testing potential networking issues somehow? Also, do you have an idea of how the performance of the library compares to some ideal?

Unfortunately, I don't think we're ready to add multipart support directly to Rocket just yet. As such, I'm moving this to 0.4. If no breaking changes are required, we might see this as a point release in 0.3.

[abonander]

Are you testing potential networking issues somehow?

What kinds of issues that aren't covered by randomized read sizes in the integration test, or fuzzing? Any I/O errors are propagated upwards.

Also, do you have an idea of how the performance of the library compares to some ideal?

What would you consider an ideal?

[gbip]

Is it planned to land soon in rocket ? I am currently lacking this feature for a project... 😄
I mean every web application needing to do file upload from a form will need to support enctype=multipart/form-data, so I think this is quite an important feature !
At the moment I'm going with the solution posted by @shepmaster (thanks by the way).

[semirix]

To anyone interested in another workaround, I devised this jQuery function to convert file uploads into base64, avoiding the need for multipart forms. The base64 string is added to a hidden element that replaces the old file input. It should work on IE10 and above but I've only tested in chrome and firefox. This will require upping the limits on all forms. This would be a reasonably viable workaround if you could set size limits per form.

$.fn.base64Upload = function () {
    return this.each(function () {
        var self = this;
        
        if (window.File && window.FileReader && window.FileList && window.Blob) {
            var inputName = $(self).attr("name");
            var encoded = $("<input>")
                .attr("type", "hidden")
                .attr("name", inputName)
                .insertAfter($(self));
    
            $(self).change(function (event) {
                var file = event.target.files[0];
                if (file) {
                    var reader = new FileReader();
                    reader.onload = function(readerEvt) {
                        var binaryString = readerEvt.target.result;
                        encoded.val(btoa(binaryString));
                    };
                    reader.readAsBinaryString(file);
                }
            });
            $(self).val("");
            $(self).removeAttr("name");
        } else {
            console.error("Cannot use base64 upload, File APIs are not supported in this browser!");
        }
    
        return self;
    });
};

Call like this:

$("input[type=file]").base64Upload();

[TheNeikos]

We're closing in on the 1 year anniversary! Is there a roadmap or list of things that need to be done for this to work? Is it just the multipart crate?

[abonander]

There is also the formdata crate if multipart is still unsatisfactory. I never got a reply to my last comment so I don't know what the blockers are. I've fixed a number of bugs in multipart since the previous discussion and I'm soon to land some significant API improvements (on master but not yet published).

[tcr-ableton]

Any news on that front? :-)

[abonander]

The core crate builds, I've been slowly fixing the examples. I remember that Rocket wanted to use multipart behind the scenes so I don't know if I should spend time building integration on my end.

[williamhgough]

Also looking forward to this. 🙊

[tforgione]

I'll put this here because I don't see it mentionned in this issue, but there is this crate rocket-multipart-form-data that seems to allow to parse multipart forms.

[xemwebe]

Since multipart does not work anymore after the latest updates with respect to async, and multipart-async does not support rocket (yet?), I raise the question again: Is anyone working on direct multipart support for rocket? Or what is the intention of the project owners with respect to multipart support? Is there any known solution that works with the current development version?

[dbrgn]

I haven't tried it yet, but I think https://github.com/rousan/multer-rs/ works with Rocket. At least I managed to successfully process incoming multipart requests in a pure-hyper application.

[xemwebe]

I haven't tried it yet, but I think https://github.com/rousan/multer-rs/ works with Rocket. At least I managed to successfully process incoming multipart requests in a pure-hyper application.

Thanks for the hint, this looks promising. I will give it a try.

[jebrosen]

Since multipart does not work anymore after the latest updates with respect to async, and multipart-async does not support rocket (yet?), I raise the question again: Is anyone working on direct multipart support for rocket?

No that I know of - and other issues have been higher priority. The biggest are compiling on stable rust and async migration, which would have taken even longer to do if multipart had to be fixed/upgraded at the same time. My own personal dislike of the multipart format is another reason I haven't seriously taken it up yet, but I firmly believe Rocket should support it.

Or what is the intention of the project owners with respect to multipart support?

Multipart support that is both correct and user-friendly is surprisingly difficult, even for seemingly "simple" cases. Consider this hypothetical API:

#[derive(FromMultipartForm)]
struct UploadForm {
    description: String,

    #[form(limit = 1*1024*1024)] // 1MB
    file1: MultipartFile,

    file2: MultipartFile,
}

#[post("/", data = "<upload_form>")]
fn upload(upload_form: UploadForm) { }

This poses several design questions:

• Where is the data for the two MultipartFiles stored? Do they implement AsyncRead, or are they in memory or on disk somewhere?
  • Supposing a streaming interface, how should rocket handle file2 coming before file1 in the request body? file2 has to be consumed before file1 is even accessible!
  • Supposing an on-disk interface, where should the files go?
• How are size limits specified and checked, both per-field and for the whole form? (This is more important for a memory- or disk-backed approach, where denial of service by memory or disk exhaustion is a concern.)
• Each field in the body can optionally have an associated Content-Type. Where and how should this be made available?
• What types can be used in this struct? Perhaps we need a new trait FromMultipartValue?

Some other rough ideas, all with different tradeoffs chosen from the above:

struct UploadForm {
    description: String,
    file1: Vec<u8>,
    file2: File,
}

// "without #[derive]". This provides marginal value over using a multipart crate directly, since the interface would be similar.
impl FromMultipartForm for UploadForm {
    // similar trait to FromForm
}

// "magic attribute". This is different from `#[derive]` because it help enforce that values could come in any order.
#[rocket::multipart_form]
impl FromMultipartForm for UploadForm {
    async fn description(&mut self, value: String) { /* ... */ }
    async fn file1(&mut self, value: MultipartFile) { /* ... */ }
    async fn file2(&mut self, value: MultipartFile) { /* ... */ }
}

Is there any known solution that works with the current development version?

There are a few options, both of which rely on the fact that data can be read as a stream of bytes and then passed to anything that will take a stream or container of bytes: 1) stream incoming data into a Vec (with a limit!), and then run a synchronous multipart parser when the data is ready. 2) Pass data.open() to an asynchronous multipart parser. Since some of the relevant traits haven't converged yet, this option might require adapters between tokio::io::AsyncRead and futures::io::AsyncRead and/or between AsyncRead and Stream<Item=Result<SomeBufferType, io::Error>>. These adapters exist in crates such as tokio-util, specifically the compat module and the BytesCodec type.

[xemwebe]

@jebrosen thanks for your detailed answer and sharing your ideas. I agree, implementing multipart in a robust, safe and user-friendly way is far from being trivial, therefore I hoped I don't have to implement my own workaround. However, I understand multipart support has currently not the highest priority.

I probably will try to use multer-rs which supports AsyncRead, but have to upgrade some other of my project's dependencies to async first before I could seriously work on that.

What do you think about an approach that supported types in the MultipartForm struct must match the ContentType of the uploaded form (with some allowed conversions)? Storage in memory vs. storing as temp file could be solved by either an attribute or special sub type.

[arnodb]

Hi, I did not have a look at multer-rs, but we have been using rocket-multipart-form-data for quite some while with rocket 0.4, and as a proof of concept I have just ported the core of this crate to multipart-async and rocket 0.5.0-dev: magiclen/rocket-multipart-form-data#9.

It is not comprehensibly tested because we still have a long way to go if we want to bump our rocket version, but it seems to be working.

If that helps...

Cheers.

[arnodb]

Hi folks, I see a controversy (soft euphemism) in another issue about the release of 0.5, stable rust, and the remaining tasks to do before 0.5 is out.

This multipart forms support is one of them.

If eventually one of myselves have some time to tackle that particular task, what would be the requirements for a PR to be accepted? I am particularly thinking about:

• dependencies: multer-rs or multipart-async or...?
• design considerations: is this comment Support Multipart Forms #106 (comment) still accurate?

[SergioBenitez]

I have already implemented this and all other 0.5 milestone issues, with the exception of #21 which was implemented by @jebrosen and @hawkw, so any efforts to implement this feature would be duplicated. Due to our standards for code quality, usability, and robustness, committing the changes upstream takes longer than you might expect. Nevertheless, you'll see this issue be closed very soon.

[arnodb]

OK, cool, I'm looking forward to seeing that.

[hamidrezakp]

I have already implemented this and all other 0.5 milestone issues, with the exception of #21 which was implemented by @jebrosen and @hawkw, so any efforts to implement this feature would be duplicated. Due to our standards for code quality, usability, and robustness, committing the changes upstream takes longer than you might expect. Nevertheless, you'll see this issue be closed very soon.

Nice, how we can use multipart in Rocket now? using version 0.5 or there is a workaround that works with version 0.4?

[jebrosen]

Nice, how we can use multipart in Rocket now? using version 0.5 or there is a workaround that works with version 0.4?

@hamidrezakp It is already possible to handle multipart forms in a manual way, by passing the request Data to a multipart/form-data parsing library - e.g. #106 (comment), which was written for rocket 0.3. For a slightly more automatic approach, rocket-multipart-form-data (mentioned in #106 (comment)) is one multipart crate that works with rocket 0.4; it also looks like the multipart crate has some integration or examples.

[hamidrezakp]

Nice, how we can use multipart in Rocket now? using version 0.5 or there is a workaround that works with version 0.4?

@hamidrezakp It is already possible to handle multipart forms in a manual way, by passing the request Data to a multipart/form-data parsing library - e.g. #106 (comment), which was written for rocket 0.3. For a slightly more automatic approach, rocket-multipart-form-data (mentioned in #106 (comment)) is one multipart crate that works with rocket 0.4; it also looks like the multipart crate has some integration or examples.

Thank you, yeah i am using multipart library and it works fine.

[TotalKrill]

Since this seems to have a well working helper crate, and is on the list for 0.5 milestones, could this maybe be pushed to 0.6? I would very much like to see a release of 0.5 expedited.

[LittleEntity]

Hello @jebrosen and @SergioBenitez

thankyou for an already amazing experience with rust web development. I had a good time reading your guide and experimenting with rocket, tera and postgresql.

I need multipart forms for a prototype. I would use a not yet stable api for the prototype. Is there a way I can use the rocket implementation right now without using a workaround which I probably will abandon soon?

I am considering myself to be a beginner rust programmer. I hope you are ok with me asking this newbie question.

[sabirmgd]

I'm trying to use the rocket client to unit-test this, any documentation on how the client can send multipart requests?

[SergioBenitez]

@sabirmgd See #1591.

[sabirmgd]

Thanks for the implementation, its easy and clean. However, I have issues that the file name and raw_name are always None

here is some code:

async fn handle(mut file: TempFile<'_>) {
    let raw_name = file.raw_name();
}

here is the curl command I used (I used Insomnia)

curl --request POST \
  --url http://localhost:3000/api/studies \
  --header 'Content-Type: multipart/form-data; boundary=---011000010111000001101001' \
  --form 'file=@C:\Users\DNDT\Desktop\images\test.txt' \
  --form =

Would there any reason that the file name returns None, is it because its Buffered?

[SergioBenitez]

Thanks for the implementation, its easy and clean. However, I have issues that the file name and raw_name are always None

here is some code:

async fn handle(mut file: TempFile<'_>) {
    let raw_name = file.raw_name();
}

Your code is not using Rocket's form support at all, but instead, reading the raw data into a TempFile. You need to use the Form guard to actually parse a form:

fn handle(file: Form<TempFile<'_>>) {
    let name = file.name();
}

There is also no reason to use raw_name().