Improve Yarn support in Nixpkgs

[pluiedev]

Problem Statement

The current state of Yarn support in Nixpkgs has been... shoddy at best, including a severe lack of documentation, lack of lockfile v2 support, year-old build failures, and (ancient) bugs, including but not limited to #140088, #198312, #184965, #144200, #139227, #284157, most of them have been stale for at least 1–2 years.

This is exacerbated by the fact that the fundamental architecture of yarn2nix has not been altered much since its inclusion in Nixpkgs 3 years ago, with literally only one commit made in the past 12 months, which would be acceptable if it were not for Yarn's rapid development as of late, especially regarding yarn-berry.

The current yarn2nix is awfully underequipped to handle codebases that have since migrated to Yarn 4.x, forcing packagers to use prebuilt upstream .deb packages (e.g. Proton Mail) or maintain a fork of an older branch (e.g. Mastodon), neither of which is at all ideal.

As more and more projects move away from Yarn v1, Nixpkgs needs to be ready to include them, and what we have today simply won't do. What's more worrying, however, is that there also appears to be a lack of people who are knowledgeable enough in Yarn to take on the challenge of updating or replacing yarn2nix with something more modern. As I scanned through the issues and PRs created related to Yarn, there seems to be a kind of aversion or self-professed lack of knowledge from prominent committers regarding Yarn, and I have to admit to being in this camp as well.

IMO, this is very problematic, because it indicates that we might just not have people who can tackle Yarn like people who tackled pnpm support in #290715. If that is the case, what should we do with all the Yarn packages we already have in Nixpkgs? They are out there, and we need to be able to build them.

Proposed Solution

Perhaps we can create a parallel to pnpm_{8,9}.fetchDeps, and add an entirely new implementation of yarn2nix that's designed for yarn-berry/Yarn 4.x. Then, we could have packages that use Yarn 1.x use yarn.mkPackage/fetchDeps and packages that use Yarn 4.x use yarn-berry.mkPackage/fetchDeps. However, the time and energy investment in creating that new implementation should not be underestimated.

[eclairevoyant]

The current state of Yarn support in Nixpkgs has been... shoddy at best,

I'd be surprised if anyone disagrees with your statement, even without listing out examples, but as with any open source project with practically no funding, someone needs sufficient time and interest, and they would also need to gain the knowledge to address it. JS ecosystem topics are far more difficult to deal with compared to probably any other language available in nixpkgs, and JS tooling seems to change every other week, hence lack of interest...

[pluiedev]

... someone needs sufficient time and interest, and they would also need to gain the knowledge to address it... JS tooling seems to change every other week, hence lack of interest...

That right there is really the key problem, isn't it? Unlike something like C, the JS ecosystem is constantly changing, with The Next New Best Thing coming out every couple of months. The reality is that unless we have someone who has domain expertise on each of these new technologies, their support in Nixpkgs would stagnate.

[eclairevoyant]

No, the key issue is that this is all volunteer work, while still expecting someone to spend dozens of hours a week maintaining JS tooling for infinite years to come. What I'm getting at is, filing this issue is acknowledging a problem - fine, that's what GH issues are for - but the way the description is phrased, it seems more like venting than a productive conversation. Unless you're willing to step up yourself, and are asking for help to get something going - then the conversation can go a different way.

It's not that unlike C imo; we have a similar lack of attention to the stdenv. Though we're fortunate to have had the stdenv and phases relatively well-defined from the past, now we have a single-digit-number of people who know what it does, and even less people to maintain it. It's similarly intimidating; few want to even try touching it. But if someone just says "stdenv sucks, pls fix" rather than stepping up, I hope you can see why that would leave a bad taste.

[pluiedev]

Yeah, I definitely could get how this can come off as venting, but the fact is that while stdenv might have a single digit of people who know how it works, Yarn currently has 0. Nada. And that's just not adequate at all.

I get that keep reiterating this point isn't very productive, and I'm not exactly interested in tackling Yarn myself since I don't even use Yarn, but this problem deserves to be pointed out and categorized so that we can explain people why it's really hard to package Yarn apps in Nixpkgs right now.

[doronbehar]

Hey @pluiedev the Yarn situation has improved quiet a bit since #318015 . We have now #324246 that tracks the packages that need to transition to the new yarn*Hooks. We also have #328544 which is pretty developed that adds yarnInstallHook. You are right though that the fact we distribute the too old Yarn v1 is not that good, but we have #254369 for that.

[pluiedev]

That's wonderful news. Thanks for all that work @doronbehar! ❤️