Question regarding streaming large bodies

[pablolagos]

Hello everyone. I know this has been asked several times, but it's been a while and there have been many changes in fasthttp.

Is there a way to copy a body directly from the source to the user without having to get it completely in memory?

This involves two aspects:

1. client: Is it able to make a request and not download the body automatically?
2. server: I have seen that there are some options, but none within the context of RequestCtx.

[pablolagos]

Hello again, We currently need this functionality and I think it could be beneficial to everyone.
Basically, the idea is:

In server

func handle(ctx *fasthttp.RequestCtx) {
     // DisableBuffering modifies fasthttp to disable headers and body buffering for this request
     ctx.DisableBuffering()
     
     // Write headers
     ctx.Response.Header.Set(.....)
     ctx.Response.Header.Set(.....)
     ctx.Response.Header.Set(.....)
     
     // Write body. Headers will be sent and cannot be modified once the body begins to be dispatched.
     // src can be any io.Reader or io.ReaderFrom or any object implementing io.WriteTo
     // The process would be unbuffered, well, actually using at most a 32KB buffer 
     // according to the current io.Copy implementation.
     // Ideal for serving large files without storing them on RAM. 
     io.Copy(ctx, src)
}

It would automatically set the Transfer-Encoding header to "chunked" and chunk the content on the fly.

I can write a PR if you think it makes sense for fasthttp

[erikdubbelboer]

It could make sense but I'm afraid it will be a lot harder to implement correctly than you think. If you can try to make a PR please do.

[missish]

Hi, is this problem solved? I am about to encounter this problem

[gaby]

@pablolagos Transferring with a 32KB buffer is not optimal, what I use is this:

buf := make([]byte, 1024*1024)
io.CopyBuffer(struct{ io.Writer }{dst}, struct{ io.Reader }{src}, buf)

This will allow copying using 1MB buffers.

[pablolagos]

Hi @gaby, I used 32Kb buffer following the Go io.Copy implementation:

https://github.com/golang/go/blob/master/src/io/io.go#L418

[gaby]

Hi @gaby, I used 32Kb buffer following the Go io.Copy implementation:

golang/go@master/src/io/io.go#L418

Yes, just not optimal for network transfers. That 32KB is more for Disk I/O.

Another option is to make it configurable, default to nil.