net/http: HTTP large file upload fails with "bufio: buffer full"

[jvehent]

Uploading large files in HTTP multipart POST requests fails with multipart: NextPart: bufio: buffer full.

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ulfr/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ulfr"
GORACE=""
GOROOT="/usr/lib/go"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build508991886=/tmp/go-build -gno-record-gcc-switches"

What did you do?

The following source code accepts uploads requests over HTTP. It works fine for files up to 15GB, but crashes for 20GB files with the error: multipart: NextPart: bufio: buffer full

package main

import (
        "log"
        "net/http"
)

func main() {
        http.HandleFunc("/upload", uploadHandler)
        http.ListenAndServe(":5050", nil)
}

func uploadHandler(w http.ResponseWriter, r *http.Request) {
        log.Printf("%s %s %s user-agent=%s", r.Method, r.Proto, r.URL.String(), r.UserAgent())
        if r.Method != http.MethodPost {
                http.Error(w, "only POST method is allowed", http.StatusBadRequest)
                return
        }
        log.Printf("parsing multipart form")
        // no more than 100MB of memory, the rest goes into /tmp
        r.ParseMultipartForm(100000000)
        file, handler, err := r.FormFile("uploadfile")
        if err != nil {
                http.Error(w, "failed to read uploadfile form field", http.StatusBadRequest)
                log.Printf("failed to read uploadfile form field: %v", err)
                return
        }
        defer file.Close()
        log.Println(handler.Filename)
}

Here's an example run with a 20GB file filed with zeroes.

$ go run multipart.go 
2018/07/31 08:29:18 POST HTTP/1.1 /upload user-agent=curl/7.59.0
2018/07/31 08:29:18 parsing multipart form
2018/07/31 08:29:43 failed to read uploadfile form field: multipart: NextPart: bufio: buffer full

upload with curl:

$ curl -vi -F uploadfile="@/home/ulfr/20gfile.bin" http://127.0.0.1:5050/upload
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5050 (#0)
> POST /upload HTTP/1.1
> Host: 127.0.0.1:5050
> User-Agent: curl/7.59.0
> Accept: */*
> Content-Length: 20971520209
> Content-Type: multipart/form-data; boundary=------------------------879658a7528de349
> Expect: 100-continue
> 

< HTTP/1.1 100 Continue

< HTTP/1.1 400 Bad Request
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Tue, 31 Jul 2018 12:29:43 GMT
< Content-Length: 37
< Connection: close
< 
failed to read uploadfile form field

* we are done reading and this is set to close, stop send
* Closing connection 0

Since my machine has 32GB of RAM, I tried boosting MAXMEMORY to 30GB by setting r.ParseMultipartForm(3000000000) but got the same results.

[meirf]

I wasn't able to reproduce this on go1.10.3 darwin/amd64.

[fraenkel]

I did reproduce the error and then it magically went away after a few runs. I am trying to see if I can cause it to happen again.

[bradfitz]

If anything, I suspect this is a "mime/multipart" bug and not a "net/http" bug.

[mvanotti]

Hi Julien!

I think the error is that you forgot to check the return value from ParseMultipartForm. When you call r.FormFile, it calls ParseMultiparForm again (because the form was not correctly parsed), and is now in an inconsistent state, making it fail.

It probably failed because mime/multipart/formdata.go writes to io.TempFile and you might have run out of space in your temp directory. The part that fails is the io.Copy writing the part body to the file.

The next time you call ParseMultipartForm, the buffer has already been partially consumed, and calls to NextPart fail to find the boundary.

I think after this kind of error (having a partially read buffer with some parts discarded), subsequent calls to ParseMultipartForm should fail immediately and with a clear error. Maybe ParseMultipartForm should support a directory where downloaded files should be stored.

[jvehent]

Hey @mvanotti ! Long time no see!
I do indeed see an error on ParseMultipartForm due to low disk space: write /tmp/multipart-629374591: no space left on device, so this might well be the issue. I will try to reproduce on a different machine that has a larger /tmp directory and close this issue if possible. Thanks!

[jvehent]

Confirmed on another system that the error I'm seeing is due to limited /tmp space, so this issue can be closed.

I'd be nice to not be limited by /tmp somehow, or at least be able to move that to a different directory, but the error message seems explicit enough (when not ignored, ah!) for the time being.

[mvanotti]

You can change the TempDir in linux by setting the TMPDIR environment variable (code link). It's going to be global to the whole program though.

If that's not OK, then setting the tmpfile folder for the multipart package would need to either allow to set it in the multipart.Reader struct or add a function something like ReadFormIntoFolder(dstFolder) which directly stores everything in a folder.

[mvanotti]

Regarding the error, I still think that the way ParseMultipartForm works is not OK. If that function fails (at least in this part), it doesn't make sense to allow it to be called again and make it do any work. It should fail immediately because there's nothing else it can do. Let's say that you have a multipart form with 2 parts, the first one is read completely but fails to be saved, then if you ignore the error, the next time you call it will give you just one part, without errors... Right? That seems like an error.

[iaburton]

Since ParseMultipartForm seems to be mostly a convenience method anyways, you could call MultipartReader and control the stream yourself, including where the data goes. Thats the more efficient solution anyways, and usually what I use for potentially large uploads (since you can stop it early and not have to download everything first).

I don't think the solution is to try and change where ParseMultipartForm puts data, but simply write a little more code using the great, already provided, alternative.