Serve /content/ routes with X-Sendfile

[HebaruSan]

Background

The /download/ route handler tries to redirect mod downloads to the CDN here (_cfg("cdn-domain") is https://spacedock.info/content/):

SpaceDock/KerbalStuff/blueprints/mods.py

Lines 549 to 552 in ae68684

	protocol = _cfg("protocol")
	cdn_domain = _cfg("cdn-domain")
	if protocol and cdn_domain:
	return redirect(protocol + '://' + cdn_domain + '/' + mod_version.download_path, code=302)

The idea is that the https://spacedock.info/content/ URLs should be handled by a fast server with access to the storage, what we've been calling AW (Apache Web Server).

Problem

CDN handling of downloads is disabled in the server config for reasons that nobody remembers (and turning it on temporarily produced unexplained timeouts and 502 statuses):

So the /content/ request comes back to the backend here:

SpaceDock/KerbalStuff/blueprints/anonymous.py

Lines 45 to 50 in ae68684

	@anonymous.route("/content/<path:path>")
	def content(path: str) -> werkzeug.wrappers.Response:
	storage = _cfg('storage')
	if not storage or not os.path.isfile(os.path.join(storage, path)):
	abort(404)
	return send_from_directory(storage + "/", path)

And the gunicorn backend is handing out big files, as not intended.

But there is yet another better method for sending files which SD already supports: Apache's XSendFile module:

SpaceDock/KerbalStuff/blueprints/mods.py

Lines 555 to 566 in ae68684

	if _cfg("use-x-accel") == 'nginx':
	response = make_response("")
	response.headers['Content-Type'] = 'application/zip'
	response.headers['Content-Disposition'] = 'attachment; filename=' + \
	os.path.basename(mod_version.download_path)
	response.headers['X-Accel-Redirect'] = '/internal/' + mod_version.download_path
	if _cfg("use-x-accel") == 'apache':
	response = make_response("")
	response.headers['Content-Type'] = 'application/zip'
	response.headers['Content-Disposition'] = 'attachment; filename=' + \
	os.path.basename(mod_version.download_path)
	response.headers['X-Sendfile'] = os.path.join(storage, mod_version.download_path)

This only happens for /download/ routes, if you set _cfg('use-x-accel') to nginx or apache and don't set _cfg("cdn-domain").

Changes

Now if you set _cfg('use-x-accel') it will also be used for /content/ routes. This should make them work better because the files will be sent by Apache instead of by gunicorn.

[DasSkelett]

The idea is that the https://spacedock.info/content/ URLs should be handled by a fast server with access to the storage. I think that's Apache Traffic Server?

Nope, that one's the Apache Web Server (AW), which is inside the production container and has access to the storage mount. ATS only handles caching, some headers, and forwarding all web requests into the right containers depending on the accessed domain.

[DasSkelett]

Works in my dev env after mounting the storage into the frontend container and setting use-x-accel=nginx (but leaving cdn-domain configured this time).
I'm also making nginx add a via header to make it clear whether it hits the /internal path via X-SendFile or not.

[DasSkelett]

Maybe we won't actually need it if we can get serving files from AW to work, but it's good to have a fallback, and simpler ways to test production-like setups locally.