What are your thoughts on cross-compilation?

[Majiir]

Question

What are your thoughts on cross-compilation support in nixpkgs? In particular, what should happen when changes in nixpkgs conflict with maintaining cross-compilation support?

Candidates I'd like to get an answer from

No response

Reminder of the Q&A rules

Please adhere to the Q&A guidelines and rules

[cafkafk]

In particular, what should happen when changes in nixpkgs conflict with maintaining cross-compilation support?

I hope it's okay that I answer this in a roundabout way, to build up the reasoning with the "easier" case of platform support.

I think that if the maintainer(s) of a package in nixpkgs have decided they want to support multiple platforms, and someone sends a PR that breaks the other platforms they have decided to support, the conflict mediation should lean towards the preferences of the maintainers. This also means that if maintainer(s) have chosen not to support some particular platform, and someone wants to add it to the package, they shouldn't expect the maintainers who haven't opted-in to the scope creep to maintain it.

Essentially, we should:

• Respect the support goals of the maintainers of packages
• Not scope creep maintainers with drive-by PRs that we're not gonna be the ones to keep running

This also means that ideally, if you want to have some package supported on some platform that the maintainers can't or wont support, you should seek to become one of the maintainers of the package, and the other maintainers should have reasonable patience for you to fix breakage related to version bumps.

I also can't help but think that in the worst case, having multiple packages for different platforms is useful in these situations, with potentially different maintainers, but this isn't a pattern used much in nixpkgs. We also don't have a way of specifying which maintainers support which platform of a package, we just assume they support all of them, which I think is a mistake.

---

I think this line of thinking extends to cross compilation, only that cross compilation is a much harder problem. If the cross compilation support is merely a drive by effort, it may unnecessarily scope creep the burden of maintainers for a fairly niche use-case, and without some maintainer that is dedicated and responsive to maintaining it at the package level and tests to ensure that it's not breaking on changes to the package, it's not always realistic.

One a more nixpkgs wide level, I think it strongly depends on the individual case, what's breaking, and what the costs and benefits of the breakage is. I don't think we should just allow breakage for progress at any cost, but if avoiding the breakage becomes too costly, or if no one steps up to maintain the cross compilation support for the affected parts, then I don't think it's reasonable to make it a blocker. That said, if there are willing contributors ready to help mitigate breakage, I'd lean towards giving them a reasonable amount of time to mitigate issues, or if the contributors breaking the cross-compilation support makes obviously unreasonable changes, to ask them to correct them.

We could create concrete guidelines and rules for this, but they would depend on the amount of people able and willing to mitigate breakage, and there would be exceptions such as security issues that would likely still take priority. But I'm sure that it's possible to get to a reasonably stable situation with cross compilation if there are people that are willing to step up and do that maintenance work, and we could also expect some amount of care to be taken to not break it, but not more than reasonable.

[getchoo]

What are your thoughts on cross-compilation support in nixpkgs?

I think it's one of the most underrated and amazing features of Nixpkgs. I was first introduced to it in Matthew Croughan's talk at May Contain Hackers 2022, where I was immediately surprised how it "just works" for most packages. It's an experience that's only close to being matched by Dockerfiles that can get very overcomplicated and require a lot of manual intervention in comparison. I really believe it should be a focus of Nixpkgs going to into the future, solely due to this massive potential

In particular, what should happen when changes in nixpkgs conflict with maintaining cross-compilation support?

I would agree with cafkafk though and say we shouldn't force this effort onto maintainers who either don't have the resources to keep things building or don't want to. This would be an undue burden, and we'd most likely end up back at square one where the package can no longer build in some cross environments. The only real exceptions to this should be in toolchains that entire cross targets depend on (i.e., LLVM, GCC), as without those building, the purpose of having the target is a bit moot.

In general, I would also want to encourage and mentor new contributors and maintainers in working on cross support for at least tier 1 platforms. It would be on a case by case basis, but I believe a majority of packages out there could have this support relatively easily -- and with better documentation and help from others, the average package maintainer could keep things building

[roberth]

I see this conflict as a shortcoming in tooling and especially in the communication enabled by the tooling.

We could document these aspects in the meta attributes. For instance, suppose I care about cross support in some package, I should be able to mark the package so that a cross build is performed in CI and I should add myself as a point of contact for help when that cross testing job fails.
I believe for most packages it may still be necessary to push forward if a solution isn't found quickly enough. The breakage will either result in more contributors helping out with cross, or the result is that cross wasn't important enough, and can be dropped.
This (broken) expectation could also be documented in meta.

I'd be happy to take suggestions for improvements, and simplifications for a first implementation of such mechanisms.

[proofconstruction]

I don't really think about cross-compilation at all, truthfully. The only time this has really mattered to me personally has been compiling stuff that will run on my Raspberry Pi, but for this I just spin up an aarch64-linux machine in AWS. Writing some Terraform to handle provisioning the build machine and running my jobs was easier than thinking about rigging up qemu to do the cross-compiling. Perhaps this comes up more when bringing up support for new platforms?

[Infinidoge]

Cross-compilation is one of the bigger strengths of Nix, in my eyes, and one of my favorite examples of Nix that I show others is how easy it is to get a cross compiler. Nix is fundamentally well suited for doing cross compilation, so I think it is an important thing to keep maintained. Every new package shouldn't be required to cross compile to every system, but packages that already cross-compile should be maintained on all of the platforms they already function on.

If a big change to Nixpkgs breaks cross compilation, then perhaps the big change needs to be rethought. (Though I will grant that how cross compilation is done currently, through splicing (see list bellow), is a well-known and well-hated abomination.)

• https://discourse.nixos.org/t/frustrations-about-splicing/49607
• Discourage/prohibit inheriting from package sets in callPackage nixpkgs#204303
• Enhancing lib.makeScope nixpkgs#68967 (comment)

[roberth]

Cross compilation works without virtualization: Nixpkgs configures the compilers to produce code for a different "host" platform than the "build" platform, which is where the compiler runs. Virtualization would only be "needed" when testing the output, which a cross-Nixpkgs does not do.

• Yet. Fix targetPackages for the purpose of native testing after cross compilation nixpkgs#347053

Cross outputs can then be tested by OfBorg when its build farm is configured to have remote builders.

I think it's amazing that Nix can take advantage of the speed of an alternate buildPlatform, but still test the execution of those build products on underresourced machines (such as slow hardware, rare/expensive hardware, or emulated VMs), making the best use of them, all within the same derivation graph, and even when building on platforms that don't have pluggable executable loaders like Linux binfmt_misc.

As a matter of practicality, the adoption of these cross changes should be a gradual process, just like the rollout of strictDeps - not an implement to whack our co-maintainers with, but something people who care about cross can help with.