Streaming large files

[xyproto]

Algernon uses too much memory when serving large files (like 500MB audio files). This might be related to caching. Please fix.

[xyproto]

Not compressing the file before checking if it would fit in the cache in a compressed state fixed this.

[epitron]

This is a great project! I really love all the built-in features.

Unfortunately, this issue has re-emerged.

It looks like the server is completely loading the file into some kind of cache or a buffer before sending it to the client: https://github.com/xyproto/algernon/blob/master/engine/handlers.go#L369

Shouldn't all static files be streamed directly from the disk to the client? (That's the default behaviour for all of the other webservers I've used.)

Additionally, files that are already compressed (eg: zip, jpg, mp4, ogg, etc.) can be streamed directly to the client without gzipping.

Could we have a function that streams static files?

[xyproto]

Thanks for reporting. I suspect that the streaming issues are in debug mode only. Could you please try the -x flag and see if it streams correctly?

[epitron]

Same issue.

Strangely, the process also balloons up to many multiples of the size of the file it's serving. When I first start algernon, the process uses 293MB of memory. Then I load a 75MB video, and the process's memory usage grows to 771MB.

Is algernon is copying the memory block as it's being passed around between functions?

Would it be a large architectural change to pass a file handle as a response, and have that stream to the client?

[xyproto]

I will create a reproducible test case and then fix this.

[xyproto]

I am able to reproduce the issue.

[xyproto]

The issue is that this line is trying to read in the entire file:

https://github.com/xyproto/datablock/blob/8c3914e5c4fe78accb948fad712c0080dfb5ded9/filecache.go#L437

I need to change the code to only read the file if:

• caching is enabled and it is under a certain size
• it's one of the supported data types that will need to be interpreted (like Lua scripts) and under a certain size

[xyproto]

Added a fix to master. Please test.

[epitron]

Just tested it:
$ ./algernon --httponly --nocache --server ~/bigfiles :8889

When I try to get a big file, this happens:

< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Last-Modified: Thu, 07 May 2015 08:36:32 GMT
< Server: Algernon 1.12.1
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Rate-Limit-Duration: 1
< X-Rate-Limit-Limit: 10.00
< X-Rate-Limit-Request-Forwarded-For: 
< X-Rate-Limit-Request-Remote-Addr: 127.0.0.1:34934
< X-Xss-Protection: 1; mode=block
< Date: Thu, 24 Jan 2019 00:55:57 GMT
< Content-Length: 18
< 
seeker can't seek

[xyproto]

Thanks for testing. I will add a better test and fix this.

[xyproto]

I updated Algernon to be able to stream large files. I tried both with your test case and the regression test.

Two new flags were also added:

--timeout=N for setting a timeout in seconds, when serving large files (but there is range support, so if a download times out, the client can continue where it left).
--largesize=N for setting a threshold for when a file is too large to be read into memory (the default is 42 MiB).

Please test. :)

[epitron]

Success!!!