Opinions about scaling down nixpkgs

[minijackson]

Question

Do you think we should scale down nixpkgs or other Nix projects, for example fewer packages, modules, or anything that has maintenance costs? What would be your plan of action?

Candidates I'd like to get an answer from

No response

Reminder of the Q&A rules

Please adhere to the Q&A guidelines and rules

[kloenk]

I think scaling down would have to be discussed a bit. Removing packages would kill one of the selling points of making it easy to add packages you need. A more viable approach could be the linux kernel way with trees. So one tree (maybe even repo if then merged in the default eval way) for modules and a tree for packages, or some other way it's split. (Though github is not to much build for that workflow, not sure how viable it would be in the end, so more a possible idea)

[cafkafk]

Do you think we should scale down nixpkgs or other Nix projects

For "other Nix projects", depends on the individual project. For instance, I think that the scale of home-manager could easily be extended, as long as there are people behind those extensions to maintain it.

For nixpkgs, as a former Guix user that came to Nix for how easily accessible contributing to nixpkgs is, and the breadth and freshness of the ecosystem, we shouldn't just scale down nixpkgs for the sake of it. After all, we are world class at having this wide and updated of an ecosystem, let's not abandon that frivolously.

...but alas, there is a maintainer burden. One solution I see here is to create a system of tiered support. That would mean that we could keep our breadth, while still not compromising or burdening the progress of nixpkgs core build systems. The ecosystem is too wide to discuss all aspects of it, so I'll focus on the one I'm the most competent with, Rust, and packages I'm very familiar with.

A tiered support system would look something like the following. For simple drive by package contributions, such as a fun program like pipes-rs that is just added once and forgotten, it would be considered a low-tiered, best-effort support level. That would mean that if e.g. it was broken after a rust update (e.g. as many rust packages were after the 1.80 update), it wouldn't be a blocker on the rust update. On the other hand, a package like eza, which has ~weekly updates and very high effort maintainership would be on a high-tiered support, meaning that it would be a potential blocker for such a rust update, given that it's maintainers are responsive and that waiting for it to update isn't something that drags on.

This would also help with other initiatives, such as creating a shared Cargo.lock file for these packages. It's easy to say that those packages that rarely see updates could all fall under a shared, best effort lock file, while those that have a much more active and responsive maintenance should be expected to keep their lockfiles up-to-date themselves.

I think that ideas like NUR were great, and obvious in a sense, and what works for Arch Linux for instance, but we've run that experiment now, and I think the conclusion is that NixOS is just too tightly coupled of a system for an alternative, still centralized user repository like that to really take off to the scale of the AUR.

A tiered system would mean that we didn't have to compromise on our massive package set, but neither on the standards of our shared nixpkgs commons, and I think it's the best way I've thought of so far to deal with this. Make it best effort, and mark how much effort each package is trying to provide, and then do our best to accommodate that.

We could even create meta attributes for the maintenance tier of a package, and have settings to block ones that aren't actively maintained. There are meta attributes coming in RFCs that could support this effort.

[getchoo]

In general, no. I would be much more in favor of automating and streamlining the maintenance of these things (like through passthru.updateScript, nixosTests, and freeform modules)

I wouldn't be against a cleanup of some parts of Nixpkgs, though. Removing unmaintained (whether by us or upstream) leaf packages would assist in reducing long term "tech debt", as well as help us save on resources with things like Hydra and the binary cache -- https://www.github.com/NixOS/nixpkgs/issues/346385 is something recent that comes to mind which follows this idea

[tomberek]

First off, I think the SC should be establishing a permanent Nixpkgs Core team to provide more focused thoughts and plans for such a question, but until then...

I'll walk through my thought process. Significantly scaling down Nixpkgs one-by-one doesn't really seem feasible. As you clean up, you will also create lots of breakage and impact users who may not be able to work around the removals. We do a bit of this during Zero Hydra Failure when for many packages we just admit defeat and mark them broken. Perhaps policies where if something has been broken for X time, it can be removed by anyone, not just the stated maintainer.

There are some experiments with splitting up Nixpkgs that are interesting. One can imagine some language ecosystems becoming their own repos and depend on a core Nixpkgs, this is one of the high-level/long-term motivations behind flakes (making this a reality would take more time+work). This is also related to the idea of having an in-repo shared set of language-specific lockfile entries, good for application and system-level consistency; but we would probably need lots of exceptions and fixes, which then re-raises the maintenance burden.

Some things we can do along the lines of "Automation over process and toil".

• emphasize and educate people about updateScript/nix-update-script/etc. I've found that many people are not aware of it nor are comfortable creating them. This can be a themed event: USE - Update Scripts Everywhere marathon! This would be a tedious thing to do en-masse, so it would be a quality of life thing to fund someone solely focus on it for a few months.
• some areas of Nixpkgs can have a policy of updateScript being required for new packages
• Leverage the nixpkgs-merge-bot. As we gain more comfortable with it, we can enhance the logging, visibility, testing, and approval permissions.

An example of a project where we scaled down was with the deprecation of NixOps. The Team Representatives group discussed and came to conclusion that we should notify people that it would no longer be maintained and is "use at your own risk". Many other similar tools have been created recent years and the need to keep it around+maintained was not worth it. Side-note: there is an effort and some funding to create a new version: NixOps4 - creative naming, I know.

[nyabinary]

I think most of the general sentiment above, but I would push for more aggressively removing broken packages and unmaintained upstreamed packages as well. I think we should keep the scope of nixpkgs and can even include nightly software if we have great automation as well (we already package some packages like that in firefox-* packages).

[phaer]

While I think more detail planning needs a team effort by nixpkgs-architecture or a new team, I think scaling down nixpkgs in terms of both code to maintain and eval time/disk size of nixos instances is a valuable goal per se.

I'd advocate for the following ideas (and be open to new ones of course ;)):

• different, easy to filter for, support tiers for packages. Core package sets like toolchains for tier 1 platforms and everything needed to boot the minimal nixos system which is bootable should...
  • ideally have at least 2 responsive, well known maintainers - or highlight that it's missing them
  • have automated update scripts
  • be covered by nixos vm tests
  • be prioritized in the upcoming security tracker efforts and in the build infra
  • maybe have a dedicated branch/channel in the long run
• support existing efforts to minimize the set of nixos modules required to evaluate a nixos system
• continue to support existing efforts on minimizing the size of nixos disk images, as those often higlight size and unecessary dependencies issues most quickly.
• Consider investment for tooling that makes it easier to "follow" a sub-set of nixpkgs. I.e. notify me on security updates or all PRs in my nixosConfigurations.

So we should allow scaling down, primarily because it enables quicker iteration on alternative distros to be spun-off nixos and better results to e.g. build disk images for appliances.

[roberth]

We have sufficient means to better deal with the volume of packages, briefly:

• Add support support tiers in meta
• Update-automation and CI improvements
• RFC 92 dynamic derivations so that lock files can be FOD-fetched instead of copied into Nixpkgs, and fetched granularly - efficiently.
• S3 replacement

As far as I know, not spending enough is currently still a bigger issue than funding.

[proofconstruction]

If anything, I think nixpkgs should be scaled up, and recent nixpkgs architecture developments like by-name are a great step towards facilitating that. As noted in my answer to #97, the issue is rather one of logistics: at all times, we must effectively allocate our resources to accomplish as many of the most impactful goals as possible.

[mschwaig]

People in the community create packages and modules mostly based on their own needs, so I think people are going to continue to maintain our modules and packages on that basis, and I don't think trying to narrow our scope on those things will help us much.

I think maybe spinning out specific language ecosystems, so that they can be still accessed through nixpkgs, but do not have to all live in the same repository might make sense, to enable more specialized workflows and innovation in language-specific packaging, but do not know how we should do this.

I agree with @tomberek that establishing a nixpkgs team would be a good idea.