chromium: cross compilation support #229265

ghost · 2023-05-01T10:05:13Z

Description of changes

This commit successfully cross-compiles chromium from buildPlatform=x86_64-linux to hostPlatform=aarch64-linux.

Other PRs included in this one (can be reviewed/merged independently):

Things done

Built on platform(s)
- x86_64-linux
- aarch64-linux (cross from x86_64-linux)
Tested basic functionality of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

pkgs/applications/audio/espeak-ng/default.nix

pkgs/applications/audio/espeak/default.nix

Artturin · 2023-05-01T21:13:04Z

separated chromium: set TERM=dumb for ninja & chromium: do not skip {pre,post}Build #229367

ghost · 2023-05-01T21:16:38Z

$ nix-instantiate . -A pkgsCross.aarch64-multiplatform.ungoogled-chromium
/nix/store/va4rvfk8i5q7pgl9g7l0wph9s5aj3lfc-ungoogled-chromium-aarch64-unknown-linux-gnu-112.0.5615.165.drv
$ nix-build /nix/store/va4rvfk8i5q7pgl9g7l0wph9s5aj3lfc-ungoogled-chromium-aarch64-unknown-linux-gnu-112.0.5615.165.drv
/nix/store/w2waiscdk40d07cd3f5x0hjca32j65gz-ungoogled-chromium-aarch64-unknown-linux-gnu-112.0.5615.165

@ofborg build pkgsCross.aarch64-multiplatform.ungoogled-chromium

xdg-utils does not cross-compile. Let's late-bind (using the runtime $PATH) in this scenario so a native-built xdg-utils can be used instead.

`llvmPackages.clang` is used in the `gnFlags` attrset, so we need to indicate explicitly which "on" platform (the build) and which "for" platform (the host) we want. This commit does that.

ungoogled-chromium is, contrary to its name, not a build of chromium. It is a patched copy of chromium's *source code*. Therefore, it needs to come from pkgsBuildBuild, because it contains python scripts which get /nix/store/.../bin/python3 patched into their shebangs.

Sometimes we access `llvmPackages` via `pkgs`, and other times via `pkgsFooBar`, so unfortunately a string (attrname) is the only way to have a single point of control over the LLVM version used for both buildPlatform and hostPlatform.

This commit successfully cross-compiles chromium from buildPlatform=x86_64-linux to hostPlatform=aarch64-linux. Other PRs in this project (can be reviewed/merged independently of this one): - #227707 - #227708 - #227710 - #227719 - #227720 - #227722 - #227723

ghost · 2023-07-12T19:38:45Z

Rebased.

michaeladler

Looks like quite the ordeal. Nice job! If it builds, then it's fine with me :)

pkgs/applications/networking/browsers/chromium/common.nix

Artturin · 2023-07-13T22:09:57Z

pkgs/applications/networking/browsers/chromium/common.nix

+      pkgsBuildBuild.pkg-config
+      libuuid
+      libpng  # `.override` used below does not splice
+    ] ++ buildInputs;


That's weird, Why is it necessary

After investigating more closely I think the comment is incorrect.

I think pkgsBuildBuild.libpng is needed for something Chromium is doing at build time, not because of anything related to splicing. I'm verifying this and will remove the comment (or the entire line) once I do.

I'm saying ++ buildInputs is weird

Chromium compiles a ton of stuff for both the hostPlatform and the buildPlatform (which, confusingly, it calls "host"... argh). For example, in order to dump out a "quick-start image" for the javascript VM it builds an entire buildPlatform version of it!

I could slog through every last dependency and figure out exactly which ones are/aren't needed on the host, but then that list would need to be maintained separately. It's definitely more than half of them.

See ea7c3166548919ba63c204f47598e8522de1c2d3

ghost · 2023-07-18T20:28:11Z

Rebased.

Artturin · 2023-08-03T22:29:35Z

Very messy but I guess that's just how browsers are...

Artturin · 2023-08-03T22:33:36Z

Merge after someone confirms that there's no build time deps included in the cross closure
nix path-info -rsh ./result | rg --invert-match "aarch"

Artturin · 2023-08-04T03:04:16Z

There's no for build deps in the output

It took 3 hours to build chromium...

ghost · 2023-08-05T05:33:06Z

There's no for build deps in the output

Should I add a disallowedReferences to ensure that none leak into the output in the future?

Artturin · 2023-08-05T05:42:21Z

There's no for build deps in the output

Should I add a disallowedReferences to ensure that none leak into the output in the future?

Sure, check how I did it in systemd.

emilylange · 2023-10-19T00:59:58Z

This increased the build steps and build time for non cross-compilation (e.g. chromium.browser) by a lot.
We are talking ~54000 vs ~85000 (and lately ~95000, showing a trend that suggests it gets even worse with each release).

It almost feels like we are building certain things twice.

Would you mind looking into this again @amjoseph-nixpkgs?

Edit: Wait. Are we cross-compiling to the same architecture if not invoked by pkgsCross?

Artturin · 2023-10-29T00:27:23Z

pkgs/applications/networking/browsers/chromium/common.nix

+      target_cpu    = chromiumRosettaStone.cpu stdenv.hostPlatform;
+      v8_target_cpu = chromiumRosettaStone.cpu stdenv.hostPlatform;
+      target_os     = chromiumRosettaStone.os  stdenv.hostPlatform;


Setting targets may be the cause of always cross-compiling

https://github.com/void-linux/void-packages/blob/7b23d90a5ea77b8d91f12c818d620e90eb710b3c/srcpkgs/chromium/template#L250-L281

Possible improvement:

chromium: do not set snapshot toolchain for native builds #264326

ghost · 2023-10-30T10:02:31Z

Edit: Wait. Are we cross-compiling to the same architecture if not invoked by pkgsCross?

Chromium has its own totally custom build system, and it's a gigantic pain in the ass. That's why chromium (and &(^&% QT) were among the very last parts of nixpkgs to support cross-compiling.

ofborg bot added the 6.topic: cross-compilation Building packages on a different platform than they will be used on label May 1, 2023

ghost requested a review from Artturin May 1, 2023 11:36

ofborg bot added the 8.has: clean-up label May 1, 2023

ofborg bot requested review from michaeladler, primeos, squalus and thefloweringash May 1, 2023 12:17

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels May 1, 2023

ghost marked this pull request as ready for review May 1, 2023 12:52

Artturin reviewed May 1, 2023

View reviewed changes

pkgs/applications/audio/espeak-ng/default.nix Outdated Show resolved Hide resolved

Artturin reviewed May 1, 2023

View reviewed changes

pkgs/applications/audio/espeak/default.nix Outdated Show resolved Hide resolved

ofborg bot requested a review from networkException May 31, 2023 05:34

ofborg bot added 10.rebuild-linux: 11-100 and removed 10.rebuild-linux: 1-10 labels May 31, 2023

Adam Joseph added 6 commits July 5, 2023 03:32

xdg-utils: mark broken if cross

84ada07

chromium: late-bind xdg-utils if broken

758bf4c

xdg-utils does not cross-compile. Let's late-bind (using the runtime $PATH) in this scenario so a native-built xdg-utils can be used instead.

chromium: take llvmPackages from pkgsBuildTarget

c25897c

`llvmPackages.clang` is used in the `gnFlags` attrset, so we need to indicate explicitly which "on" platform (the build) and which "for" platform (the host) we want. This commit does that.

michaeladler approved these changes Jul 13, 2023

View reviewed changes

Artturin reviewed Jul 13, 2023

View reviewed changes

pkgs/applications/networking/browsers/chromium/common.nix Outdated Show resolved Hide resolved

Artturin reviewed Jul 13, 2023

View reviewed changes

ofborg bot requested a review from michaeladler July 14, 2023 06:04

ghost requested a review from lorenz July 18, 2023 20:28

ghost mentioned this pull request Jul 18, 2023

Init: chromium darwin #244152

Closed

12 tasks

Artturin merged commit 4055d18 into NixOS:master Aug 4, 2023

ghost deleted the pr/chromium/cross/all branch August 5, 2023 05:32

This was referenced Aug 5, 2023

chromium: (cross) invoke ungoogled-chromium via buildPackages #227710

Closed

chromium: (cross) take llvmPackages from pkgsBuildTarget #227719

Closed

Artturin reviewed Oct 29, 2023

View reviewed changes

ghost mentioned this pull request Oct 30, 2023

chromium: do not set snapshot toolchain for native builds #264326

Closed

13 tasks

emilylange mentioned this pull request Dec 10, 2023

chromium: fix increased build time for non-cross-compilation builds, add ungoogled- prefix to chromium-unwrapped, move stray patches to right location #273435

Merged

13 tasks

Artturin mentioned this pull request Jan 29, 2024

List of @amjoseph-nixpkgs Pull requests before account removal #284796

Open

emilylange mentioned this pull request Jun 10, 2024

[staging-next] electron: fix build #318857

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chromium: cross compilation support #229265

chromium: cross compilation support #229265

ghost commented May 1, 2023 •

edited by ghost

Loading

Artturin commented May 1, 2023

ghost commented May 1, 2023

ghost commented Jul 12, 2023

michaeladler left a comment

Artturin Jul 13, 2023

ghost Jul 14, 2023

Artturin Jul 14, 2023

ghost Jul 14, 2023

ghost Jul 14, 2023

ghost commented Jul 18, 2023

Artturin commented Aug 3, 2023

Artturin commented Aug 3, 2023 •

edited

Loading

Artturin commented Aug 4, 2023 •

edited

Loading

ghost commented Aug 5, 2023

Artturin commented Aug 5, 2023

emilylange commented Oct 19, 2023 •

edited

Loading

Artturin Oct 29, 2023

ghost Oct 30, 2023

ghost commented Oct 30, 2023

chromium: cross compilation support #229265

chromium: cross compilation support #229265

Conversation

ghost commented May 1, 2023 • edited by ghost Loading

Description of changes

Things done

Artturin commented May 1, 2023

ghost commented May 1, 2023

ghost commented Jul 12, 2023

michaeladler left a comment

Choose a reason for hiding this comment

Artturin Jul 13, 2023

Choose a reason for hiding this comment

ghost Jul 14, 2023

Choose a reason for hiding this comment

Artturin Jul 14, 2023

Choose a reason for hiding this comment

ghost Jul 14, 2023

Choose a reason for hiding this comment

ghost Jul 14, 2023

Choose a reason for hiding this comment

ghost commented Jul 18, 2023

Artturin commented Aug 3, 2023

Artturin commented Aug 3, 2023 • edited Loading

Artturin commented Aug 4, 2023 • edited Loading

ghost commented Aug 5, 2023

Artturin commented Aug 5, 2023

emilylange commented Oct 19, 2023 • edited Loading

Artturin Oct 29, 2023

Choose a reason for hiding this comment

ghost Oct 30, 2023

Choose a reason for hiding this comment

ghost commented Oct 30, 2023

ghost commented May 1, 2023 •

edited by ghost

Loading

Artturin commented Aug 3, 2023 •

edited

Loading

Artturin commented Aug 4, 2023 •

edited

Loading

emilylange commented Oct 19, 2023 •

edited

Loading