stdenv: begin work on 256b bootstrap i.e. Trusting Trust

[emilytrau]

Description of changes

This starts work on building a bootstrap toolchain from a 256-byte binary seed. The aim is to build up to, and hopefully replace, stdenv's trusted bootstrap-tools bundle. It would also have the advantage of reducing the need for manual action by maintainers whenever a bootstrap update is required.

By building the new toolchain from-the-bottom-up it should be quicker to incrementally build, review, and merge smaller additions without mass rebuilds.

This PR implements packages for stage0-posix, mes, and tinycc. I've decided to limit the scope here to not make it too large for review.

Only i686-linux has been implemented for now. To test nix-build --system i686-linux . -A trusting-trust.tinycc-with-mes-libc

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[alyssais]

Thank you so much for looking into this.

[SuperSandro2000]

The biggest problem I see with this is, that we must make it maintainable. There need to be good comments explaining why things were done like they were done and helping others fix future problems.

[emilytrau]

I made an attempt at refactoring the meta attribute checking/augmentation so it can be reused outside mkDerivation with 21aaabc. I feel the change should be /ok/ but would love to hear if anyone has better ideas for implementing it.

EDIT: scratch that ofborg eval fails :(

[roconnor]

Hi. I happen to have been working on this same project last week.

I've hastily put together a repo so you folks can view my work. See https://github.com/roconnor/nixpkgs-bootstrap. So far I've gone as far as building bash-2.05b musl-1.1.24.

I wasn't really ready to showcase this yet, so it is still unpolished and has no documentation. My packages for tinycc and lower are not as refined as yours. In particular, I wasn't really aware of your builtins.placeholder trick for bootstrapping.

Another difference is that my project is deliberately designed to be independent of nixpkgs, though with the idea that it could be integrated into nixpkgs without difficulty.

[Artturin]

Fixes #123095 ?

[Artturin]

Split out stdenv: factor out meta attr augmentation for reusability #228759 (eval failure fixed there)

[trofi]

Is there a 1-pager that explains what the end state of bootstrap should look like for nixpkgs (a few crucial package dependency arrows for stdenv)? If not, would it be reasonable to ask to outline it in PR? Ideally compared to the current bootstrap we have today.

I have a few idle questions that might steer writing a short overview:

1. How many steps do we need to get gcc-12 built?
2. Is the result practical for tweaking and porting to other architectures and OSes?
3. Is it expected to be an experiment to tinker from time to time? Or it's expected to be the primary way to build nixpkgs "soon" (say, in a year).
4. Do we expect (many) other upstream linux distributions to use it or will it always be a burden to maintain in 2-3 downstream distributions? Toolchains are not easy to maintain by a small group of people.

[roconnor]

https://man.sr.ht/~oriansj/bootstrappable/live-bootstrap.md illustrates a 100 step process to get to gcc 4.7.4. I've been working through it, and it doesn't seem that bad. Certainly the live-bootstrap folks have already done all the hard work. The live-bootstrap project does actually all the way to gcc 12, but Nixpkg's bootstrap-tools currently starts at gcc 8.3, so arguably that is as far as we need to go (along with gernerating other tooling in bootstrap-tools).

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-supply-chain-security-project/34345/5

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixpkgs-supply-chain-security-project/34345/8

[sdht0]

Hi. Just curious. What are the remaining steps before this can replace bootstrap-files?

Also, how easy or hard is it to extend this to x86_64? From what I understand, guix is already able to do that?

[Atemu]

Btw @emilytrau if there's anything we can help you with getting full-source bootstrap over the finish line, let us know!

[Ericson2314]

Yeah it would be really cool to have this for 24.05!

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/aux-foundational-packages/46707/4