Current GNOME ISO exceeds Hydra output limit on Hydra

[sternenseemann]

Describe the bug

The GNOME ISO job currently fails bcause it's output exceeds the configured output limit on Hydra: https://hydra.nixos.org/build/167219646, eventually preventing nixos-unstable from advancing.

I'm not sure what caused this, presumably the last staging-next rotation?

cc @NixOS/gnome

[jtojnar]

See also #141521 (comment)

[andersk]

Here are some suspicious -dev output dependencies in the GNOME ISO closure, found with nix-tree:

• stdenv-linux → gcc-wrapper-10.3.0 → gcc-10.3.0 → glibc-2.33-108-dev
  • Maybe this one’s intentional.
• exiv2-0.27.5 → gcc-10.3.0 → glibc-2.33-108-dev
  • I found this to be a result of -D_GLIBCXX_ASSERTIONS. See also exiv2: Add lib/static outputs, remove reference to gcc #131868.
• gst-editing-services-1.20.0 → gst-devtools-1.20.0
• packagekit-1.2.5pre → nix-2.6.0-dev
• net-snmp-5.9.1-lib → openssl-1.1.1m-dev
• pipewire-media-session-0.4.1 → pipewire-0.3.45-dev
• python3.9-brotlicffi-1.0.9.2 → python3.9-cffi-1.15.0-dev

[andersk]

Some selected large packages that jump out at me:

• linux-firmware-20211216 is 791 MB; might it make sense to try to split that up somehow and drop less important firmware from the ISO?
• speech-dispatcher-0.11.1 pulls in multiple speech systems, including mbrola-3.3 at 677 MB and flite-2.2 at 64 MB; do we need them all?
• There are two copies of webkitgtk-2.34.5 with different hashes, at 124 MB each; one of them is used only by yelp-41.2.
  • yelp: Use webkit2gtk-4.0 ABI #160278
• icu4c-67.1 and icu4c-70.1 are both 36 MB; the former is used only by spidermonkey-78.15.0
• There are two copies of grub-2.06 with different hashes, at 32 MB each, both pulled in directly by system-path.
• There are two copies of openblas-0.3.19 with different hashes, at 29 MB each, one pulled in by opencv-4.5.4, and the other by blas-3 and lapack-3.
• systemd-minimal-249.7 is 12 MB and should be redundant with systemd-249.7.
  • Yeah, I know this is there to solve an annoying circular build dependency issue—I was responsible for the hack that led to the current solution.

[andersk]

• mesa-21.3.5-drivers has a bunch of large identical duplicate libraries in /nix/store/2gl456ql1g1pj9qwkgkpjajknc1j702h-mesa-21.3.5-drivers/lib/dri that are not symlinked or hard-linked.

  • crocus_dri.so = iris_dri.so = kms_swrast_dri.so = nouveau_dri.so = r300_dri.so = r600_dri.so = radeonsi_dri.so = swrast_dri.so = virtio_gpu_dri.so = vmwgfx_dri.so: 29 MB each
  • i830_dri.so = i915_dri.so = i965_dri.so = nouveau_vieux_dri.so = r200_dri.so = radeon_dri.so: 19 MB each
  • nouveau_drv_video.so = r600_drv_video.so = radeonsi_drv_video.so: 15 MB each

  Symlinking would save some 381 MB (though maybe SquashFS already does its own deduplication?).

  mesa.drivers: Convert hard links to symlinks #160267

[jtojnar]

• pipewire-media-session-0.4.1 → pipewire-0.3.45-dev
• python3.9-brotlicffi-1.0.9.2 → python3.9-cffi-1.15.0-dev

I tried removing these references but the size changes were insignificant, IIRC.

speech-dispatcher-0.11.1 pulls in multiple speech systems, including mbrola-3.3 at 677 MB and flite-2.2 at 64 MB; do we need them all?

Presumably, different implementations have different quality for different languages. Mbrola was specifically requested.

There are two copies of webkitgtk-2.34.5 with different hashes, at 124 MB each; one of them is used only by yelp-41.2.

This is a consequence of libsoup-3.0 changing ABI so webkitgtk-4.0 linked against libsoup-2.0 cannot be used in packages linked against libsoup-3.0. However, most of those (including Yelp) will still build with webkitgtk-4.0 and libsoup-2.0 so we could switch to that for now.

[jtojnar]

By the way, as more and more apps switch to GTK 4, we will need to ship webkitgtk_5_0 ABI variant so there will need to be two webkit versions anyway (unless everything manages to switch this cycle).

[andersk]

strip would shave 27 MB from libwebkit2gtk-4.1.so.0.0.5 and libjavascriptcoregtk-4.1.so.0.0.5. We could probably save truckloads of space with some stdenv-wide equivalent of Debian’s dh_strip.

[jtojnar]

Would not that also destroy debugging symbols? We want those for webkit because having people rebuild webkit to report crashes is not really feasible.

[andersk]

webkitgtk has separateDebugInfo = stdenv.isLinux. I don’t know exactly what needs to be preserved for separate debug info to work, but I do know that Debian also uses separate debug info, and its webkitgtk libraries are much smaller than ours, and cannot be further reduced by strip.

[andersk]

https://stackoverflow.com/q/46197810/115030 addresses exactly that question, and suggests that we should be able to safely use --strip-unneeded.

• separateDebugInfo: Use --strip-unneeded #160259

[sternenseemann]

I feel like the long term solution to this problem may very well be having a way to allow bigger outputs on Hydra for certain jobs. Having a big graphical live system may very well be a fact of life.

[dasJ]

@sternenseemann I think it's worthwile to keep it below 4.7G so it still be burned onto a regular DVD. Burning it onto a CD seems pointless at this point because 700M is pretty much out of reach by now :D

[06kellyjac]

I think a small image should be a goal but yes a graphical live system can only be so small.

POP!_OS is 2.47GB
elementary OS is 2.5GB
Ubuntu is 2.9GB
Linux Mint is 2.1GB
Manjaro XFCE which I expected to be fairly small is a wopping 3.4GB and their minimal XFCE edition is 2.6GB
Endevor OS is 1.9GB

[vcunat]

Burning it onto a CD seems pointless at this point because 700M is pretty much out of reach by now :D

The -minimal ISO is around 0.8G, so maybe there it isn't out of reach, but I don't think CDs are really common anymore. As for flash drives, the interesting limits often are around powers of two.

[mannp]

Could someone have a look at NixOS/nixos-org-configurations#206 and decide whether it is good idea or not?

Or perhaps move it so more important security updates and patches can be released.

Having patches released very quickly, and then waiting a week for the unstable channel update because of this failed job, seems counterintuitive :-(

[vcunat]

The -small channels were created (years ago) to get security fixes quickly. Those have been unaffected. They tend to update at least daily, quite reliably.

[mannp]

They do, to which I have tried, but they only include a small subset of the packages, including no gnome.

Am I able to merge channels for security updates and general apps together? As i'd understood, it was one or the other.

It seems perhaps more pragmatic to move longer term challenges into their own lower priority channel.

[vcunat]

The -small channels block on a small set of builds, but everything shares the same cache.nixos.org. So each binary is available in cache immediately after it gets built, and you'll get them regardless of which channel you follow.

[vcunat]

After Graham resolving some resulting complications, we got a >2G ISO build: https://hydra.nixos.org/build/167219646

[bobvanderlinden]

Awesome! Thanks for checking. I was still looking https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents, but that is lower in the queue I guess.

[drupol]

After Graham resolving some resulting complications, we got a >2G ISO build: https://hydra.nixos.org/build/167219646

Greater than 2G ? Or smaller?

[vcunat]

Well, we've been fighting this from more sides. Limit got raised, so the week-old ISO build succeeded when retried.

File size: 2160033792 bytes (2059.97 MiB)

[vcunat]

The tested job is also at the top of the queue, but it needs many builds to complete, which is probably why it takes time.

[zowoq]

As this isn't blocking the channel now I've removed the label so this issue doesn't appear at the top of https://status.nixos.org/

[vcunat]

The nixos-unstable channel updated 🎉 (to a commit from 28 hours ago) Thanks everyone who's helped with this. Some suggestions from above would still be nice to resolve.

[bobvanderlinden]

Some of the sub-problems have also been moved to separate issues by @andersk. I think the main goal of this issue is now solved. Can this issue be closed?

Should there be an issue for moving ISO generation away from Hydra channels into something separate?

[vcunat]

Separate? I can't recall what you mean. It's always been running on a separate machine, I believe.

[bobvanderlinden]

Ah, I was referring to separating the generation of the ISO like discussed here #161070 or suggested here.

[LunNova]

There are two copies of openblas-0.3.19 with different hashes, at 29 MB each, one pulled in by opencv-4.5.4, and the other by blas-3 and lapack-3.

Should we even have opencv around? It seems like it only gets pulled in by gst-plugins-bad.

Looks like that supports building without opencv as a dep.

Could either drop gst-plugins-bad or disable opencv in it by default?

[K900]

Should we close this one?

[vcunat]

It's not a blocker anymore, though in my eyes there's at least one point above worth looking into:

• speech-dispatcher-0.11.1 pulls in multiple speech systems, including mbrola-3.3 at 677 MB and flite-2.2 at 64 MB; do we need them all?

[K900]

Maybe this is better handled as a separate issue?

[vcunat]

Hmm, picking up the parts that are still relevant and posting them into a new issue? Maybe.

[Madouura]

This has been affecting the ROCm builds a bit, particularly concerning rocm-llvm (with clang-tools-extra), miopen (with KDBs), and rocfft. (It's just a gigantic .so, with everything built)
I'm not sure how realistic it is to ask this, or the reasons why we would not want to do this, but raising the limit to around 5GiB would be ideal concerning the ROCm packages.