nginx: add v2 get_source endpoint
#2521
Conversation
We currently ignore any trailing garbage after a matched URL. Return 404 instead to improve hygiene. I'm not aware of anything currently sending trailing garbage, but if such code exists this will obviously break it.
wrapdb.mesonbuild.com abstracts over the GitHub URLs for wrap files and patch zips, but not for mirrored sources: create_release.py generates source_fallback_urls which point directly to the wrap's GitHub release. As a result, GitHub isn't just an implementation detail: we can't retroactively add additional mirrors, and the existing mirror will break if we ever have to move off GitHub. Add a get_source endpoint similar to get_patch so create_release.py can start using it.
|
The new URLs look like: https://wrapdb.mesonbuild.com/v2/zlib_1.3.1-3/get_source/zlib-1.3.1.tar.gz |
|
The point in using GitHub as fallback is also in case wrapdb.mesonbuild.com is down, IIRC. |
|
What do you need from me here? Just deploy the new config on the wrapdb servers? |
For the wrap itself it's not a big deal because projects commit them. But it's a fair point that we don't write Maybe we should add That said, we can merge (and deploy) this, we can have the discussion about changing the wrap generator in another PR. |
|
Oh, that's a good thought. I'll add |
The point in uploading copies of source tarballs to GitHub was certainly not to handle the case where WrapDB.mesonbuild.com is down -- it was to handle the case where upstream websites are down. The source url never passed through the wrapdb redirector to begin with. That being said I think I agree that this change doesn't seem entirely necessary -- there's no benefit as far as I can tell for this indirection. If we actually wanted to...
... do this, then it wouldn't matter because all historical fallback sources would still exist on GitHub (unless we chose to salt the earth on our way out by deleting all organization assets from GitHub instead of simply archiving and making it all read only). For the *.wrap files, it makes sense to ensure that our redirector service is the single source of truth, because it's how we disseminate new versions. For the patch_url overlay archives I'm not sure it matters either, though it is easier to have the "primary" functionality of a wrap go through our own endpoints as there's less to change if/when we migrate again. But for the fallback sources, we are literally just using GitHub as an additional CDN / backup mirror, it's not even critical to begin with, and if we did switch service providers I think we would need to reevaluate whether we want to (or can) provide additional backup mirroring at all. |
|
You're right that it isn't entirely necessary; it's future-proofing. With GitHub as it is today I don't expect we'd ever have a reason to switch, but for an unknown future badly-behaved GitHub (maybe one that charges for storage?) it seems useful to control our own endpoints. Since we already have a mechanism for that, I figure we might as well use it. (Also, at some point I still want to pursue using the source mirror by default, to reduce the load on upstream servers. Controlling our own mirror URL would be more important in that eventuality. Not relevant to this PR, but for the record: that change presupposes a new Meson option allowing the canonical upstream URL to be used instead.) |
Yes. :P (For Gentoo, it's entirely user defined via |
|
Followup in #2532. |
|
Deployed on both wrapdb machines now. |
|
Thanks! |
4 participants
wrapdb.mesonbuild.com abstracts over the GitHub URLs for wrap files and patch zips, but not for mirrored sources:
create_release.pygeneratessource_fallback_urls which point directly to the wrap's GitHub release. As a result, GitHub isn't just an implementation detail: we can't retroactively add additional mirrors, and the existing mirror will break if we ever have to move off GitHub. Add aget_sourceendpoint similar toget_patchsocreate_release.pycan start using it.While we're here, disallow trailing garbage in v2 URLs to improve hygiene. (I'm not aware of anything currently sending trailing garbage, but if such code exists this will obviously break it.)
cc @tp-m