Switch to .xz by default for SDK downloads #1281
Conversation
|
I manually uploaded an |
|
You can see it download in the .xz file here: https://app.circleci.com/pipelines/github/emscripten-core/emsdk/2059/workflows/2832797a-6de0-4179-88ba-baff916ad11f/jobs/12259?invite=true#step-105-7623_256 |
This is a bit of a hack but I can't think of another way to do it. Basically when downloading SDKs, we first try the new `.xz` extension. If that fails, we fall back to the old `.tbz2`. Both these first two download attempts we run in "silent" mode. If both of them fail we re-run the original request in non-silent mode so that the error message will always contain the original `.xz` extension. See #1235
| @@ -34,8 +34,8 @@ | |||
| "version": "%releases-tag%", | |||
| "bitness": 64, | |||
| "arch": "x86_64", | |||
| "linux_url": "https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/%releases-tag%/wasm-binaries.tbz2", | |||
| "macos_url": "https://storage.googleapis.com/webassembly/emscripten-releases-builds/mac/%releases-tag%/wasm-binaries.tbz2", | |||
| "linux_url": "https://storage.googleapis.com/webassembly/emscripten-releases-builds/linux/%releases-tag%/wasm-binaries.tar.xz", | |||
There was a problem hiding this comment.
I wonder if it's worth leaving a comment here that the .tbz2 equivalents to these URLs are also tried if xz is not fuond.
There was a problem hiding this comment.
json doesn't support comments 😭
There was a problem hiding this comment.
Oh right, I always forget that 😿
| # `.xz`, but we can't tell from the version/url which one to use, so | ||
| # try one and then fall back to the other. | ||
| success = False | ||
| if 'wasm-binaries' in archive and os.path.splitext(archive)[1] == '.xz': |
There was a problem hiding this comment.
I guess this will also cause a retry for any other kind of download failure (first a silent failure, then a retry with a potentially loud failure), but in the successful case of fetching an xz, there will be only one request, right?
|
Relatedly, I wonder if we can do any better on Windows. I don't know of any other archive formats natively supported by Windows, do you? I wonder if it's worth bundling something with emsdk, it makes a pretty big compression difference. |
|
(oh, maybe you should mention that this only affects Mac and Linux. If we could do something for windows too, it could still have a big impact). |
Mention where? |
|
I guess I meant in the commit description; but as mentioned this probably warrants a changelog entry and maybe an announcement to emscripten-discuss (the fact that most users need to update emsdk to see new versions makes this less of a big deal I guess, but still maybe worth doing). |
|
Will do |
This is a bit of a hack but I can't think of another way to do it. Basically when downloading SDKs, we first try the new `.xz` extension. If that fails, we fall back to the old `.tbz2`. Both these first two download attempts we run in "silent" mode. If both of them fail we re-run the original request in non-silent mode so that the error message will always contain the original `.xz` extension. See emscripten-core#1235
This is a bit of a hack but I can't think of another way to do it. Basically when downloading SDKs, we first try the new
.xzextension. If that fails, we fall back to the old.tbz2. Both these first two download attempts we run in "silent" mode. If both of them fail we re-run the original request in non-silent mode so that the error message will always contain the original.xzextension.See #1235