package fetching: support .tar.zst archives

[dweiller]

Resolves #14299.

I have tested this with zig fetch on a local .tar.zst archive, but I'm not aware of any code forges online that produce zstandard compressed archives. As I haven't been able to test against code forges I have not added any http content type recognition - should I just add application/zstd and/or application/tar+zstd or leave it until someone needs it in the future?

[andrewrk]

Thanks!

Yes please add the proper mime type detection as well. You should be able to find an official mime type database somewhere; no need to find an example web server, just support the standard.

[andrewrk]

Can you also share how this affects compiler binary size? The zstd implementation looks a bit hefty. Since it's a less commonly used format, it might make sense to rely on a fetch plugin for this one if it adds too much bloat to the compiler.

[dweiller]

Can you also share how this affects compiler binary size? The zstd implementation looks a bit hefty. Since it's a less commonly used format, it might make sense to rely on a fetch plugin for this one if it adds too much bloat to the compiler.

Looks like the difference is exactly 26KiB (before the couple extra bytes from adding zstd mime type):

> du -b zig-zstd-fetch zig-ca690ffa2
207191680	zig-zstd-fetch
207165056	zig-ca690ffa2
> echo "207191680 - 207165056" | bc
26624
> echo "1024 * 26" | bc
26624

These numbers are for ReleaseFast (compiled via cmake with -DCMAKE_BUILD_TYPE=Release)

[andrewrk]

I think those sizes are unstripped, yeah? You'll probably see an even smaller delta if you run strip on them to remove debug info.

[dweiller]

I think it would have been unstripped - is there a particular invocation of strip you recommend? Running each through strip -s removed only 8 bytes from each

[andrewrk]

I got the same as you and I was briefly surprised, however, I figured it out. We both have configured cmake with -DCMAKE_BUILD_TYPE=Release as opposed to -DCMAKE_BUILD_TYPE=RelWithDebInfo so it ends up passing -Dstrip to zig build when self-hosting.

So - never mind, I think your original sizes are already stripped. Thanks for checking!

[dweiller]

I've added detection for the application/zstd content type and rebased. The latest zig nightly download is smaller by about 47MiB than what I get compiling on my machine, with fewer sections in the elf file, so a smaller compiler can be produced which might change the delta.

[dweiller]

Looks like the x86-64_windows-debug CI job hit 6 hours and timed out.

[truemedian]

Can you also share how this affects compiler binary size?

Just a note: the compiler already depends on the zstd implementation through std.http (for http decompression), so it shouldn't bloat size much if at all.

[andrewrk]

Gotcha, thanks for pointing that out!