cmd/link: support for zstd-compressed DWARF sections

[stapelberg]

I recently read a blog post by @MaskRay comparing different compression algorithms for DWARF: https://maskray.me/blog/2022-09-09-zstd-compressed-debug-sections

The blog post concludes that zstd is the best choice (as it is in many other scenarios), and the author has done some work on establishing ELFCOMPRESS_ZSTD.

I don’t know if it’s too early, or if anything else is preventing us from doing so, but wanted to file this issue to track support for using zstd to make Go DWARF sections smaller and faster to write and read.

There is a good native Go implementation of zstd available: https://pkg.go.dev/github.com/klauspost/compress/zstd

[dsnet]

Would it make sense to consider a standard library "compress/zstd" package first before considering this?

[stapelberg]

Would it make sense to consider a standard library "encoding/zstd" package first before considering this?

I don’t think we need to gate landing zstd DWARF on having a zstd package in the standard library — we could just import it into internal, or vendor it, or whatever is the currently preferred approach :)

But, having encoding/zstd in the standard library would certainly be a welcome addition IMHO :)

Can you open a separate issue for that please?

[MaskRay]

@mengzhuo added ELFCOMPRESS_ZLIB support to cmd/link and may be interested:)

[Foxboron]

I'd like to just point out that elfutils without support for ZSTD is going to be painfull for downstream as we can't make debug packages out of it. I'd rather have us disable compression and DWARF headers by default.

[stapelberg]

Sounds like it should be behind a flag for now, or added only once elfutils support for zstd is more widespread.

[cherrymui]

Do the debuggers support this format? GDB? LLDB? Delve? Thanks.

cc @aarzilli @thanm

[dsnet]

Does the runtime ever look at the DWARF section, will we need to link in a zstd decompressor?

[cherrymui]

@dsnet the runtime doesn't look at the DWARF sections.

[mengzhuo]

@mengzhuo added ELFCOMPRESS_ZLIB support to cmd/link and may be interested:)

I'm glad to do that :) but as @dsnet suggests it looks like we need a encoding/zstd in the first place.

cc @rsc

[aarzilli]

Do the debuggers support this format? GDB? LLDB? Delve? Thanks.

cc @aarzilli @thanm

According to the blog post, it was proposed to SysV gABI in june of this year and it is supported by approximately nothing. Delve will support it when debug/elf will support it, so I think it should be added there first (or at least at the same time). Other consideration:

• I think @dsnet is right that there should be a compress/zstd first, it would be weird if the standard library supported decompressing zstd but only if it's inside an elf container
• this is all about elf compressed sections, it does not apply to zdebug sections, which are emitted for mach-o and PE, and will need to stay on zlib
• given how slow binutils moves in LTS distributions this likely can not be the default (or only) option for several years
• the list of feature requests at the end of the article is highly incomplete, for example bloaty (which the article even uses) will have to be changed to support zstd compressed elf sections
• it probably won't be a problem but compression speed is also a factor for cmd/link.

[MaskRay]

FWIW clang/ld.lld/llvm-objcopy has supported zstd for ~2 weeks now. I just landed zstd support to binutils and gdb https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=2cac01e3ffff74898c54fa5e6418817f5578adb6 :)

[znkr]

IIUC, cgo is reading object files created by a C compiler, using debug/elf. Clang using zstd for compression is going to cause an incompatibility should it switch to zstd by default.

[Foxboron]

The default is controlled by a compile-time flag which should be selected by the distro, along with dwarf compression being off by default. I don't think it's too much to worry about yet?

[znkr]

I agree, I just wanted to make sure to mention that Go also consumes object files as another reason to support zstd. :)

[marxin]

I would like to point out that the latest binutils release 2.40 supports the zstd compression. Similarly master branch of elfutils (waiting for a release). So please consider adding support for cgo soon.

[gopherbot]

Change https://go.dev/cl/473356 mentions this issue: internal/zstd: new internal package for zstd decompression

[gopherbot]

Change https://go.dev/cl/473256 mentions this issue: debug/elf: support zstd compression

[znkr]

IIUC, zstd is not used for -compressdwarf yet and thus can't be used to compress DWARF sections in Go binaries. Was the issue intentionally closed without adding support in cmd/link?

[ianlancetaylor]

Sorry, I think I misunderstood this issue. The immediate problem of handling code generated by newer tools is addressed. But you're right that cmd/link still does not compress DWARF with zstd. That is a bigger hill to climb. With our current approach to vendoring in the standard library, I don't know that we want to vendor in a large package like github.com/klauspost/compress/zstd, which includes assembly code. Anyhow, reopening.

[znkr]

I think the issue was conflating two problems. Support for zstd when compiling Go and interop with C/C++ toolchains using zstd. The latter should be fixed now. Thank you. :)

For supporting zstd in Go itself: It might also be interesting to compress dwarf sections in object files as well and not just in the linker. If I find the time I might hack something together to get some data.

[cherrymui]

For supporting zstd in Go itself: It might also be interesting to compress dwarf sections in object files as well and not just in the linker.

The Go object file doesn't have DWARF sections. It just has a bunch of symbols, some of which may be discarded at link time. And there are some DWARF data generated at link time. Maybe I missed something, but I'm not sure we want to do DWARF compression in object files.

Also, how is this related to zstd? Whether to compress in object files or at link time, and which compression algorithm to use look pretty orthogonal to me. Or zstd has anything special that prefers one or the other?

[znkr]

My bad. I don’t really understand the whole process very well. What I am wondering, and this should likely be a new issue, is if it’s possible to compress debug sections in files similar to clang’s -gz flag. zstd is interesting here because it provides the right tradeoffs between compression speed and size and that could improve build speeds especially with larger builds: https://maskray.me/blog/2022-09-09-zstd-compressed-debug-sections

[cherrymui]

Thanks. By "compress dwarf sections in object files" maybe you mean in external linking mode (pre-)compressing the DWARF sections in the go.o file that the Go linker passes to the C linker? Maybe we could do that. We probably could also do that for other compression algorithms. Not sure if there is anything zstd specific.

[ianlancetaylor]

Just to be clear, in internal linking mode cmd/link already compresses the DWARF sections using zlib. So one aspect of this issue would be to compress them using zstd.

[znkr]

I think where I got things wrong is assuming that Go object files have the same format as clang and GCC formats. IIUC, that's not the case and Go has a custom format. @cherrymui, from your comment, I am guessing that there is no explicit debug info section in Go object files that could be compressed. If that's the case, my previous comment about compressing DWARF sections in Go object files is based on me assuming that there is such a thing in Go, because it exists in C.

That said, compressing the debug information in object files for external linking might be interesting. I'll see if I can get some data on that.

[znkr]

I performed some measurements by using objcopy --compress-debug-sections=zstd on intermediate object files. The linking speed of the external linker was about the same, but the file sizes shrank quite significantly. In one case from 3.2 GiB to 2.2 GiB.

[gopherbot]

Change https://go.dev/cl/499616 mentions this issue: doc/go1.21: mention debug/elf changes