Migrate away from yarn v1

[sdirix]

At the moment we're using yarn v1 for package management in Theia's mono repository.

Yarn v1 is quite old and has a lot of known bugs. Even the yarn maintainers themselves advocate switching to the latest versions of yarn (v3 and/or v4) which uses different package management concepts by default. Since the introduction of yarn, the default Node package manager npm caught up in features and stability and is generally preferred. However there are also alternatives like pnpm which again uses a different package management concept.

It should be evaluated which package manager is the new best fit for Theia. Criteria are at least popularity, maturity, feature set, outlook in the future and migration effort for the Theia repositories and potentially downstream consumers. The choice should be validated with a POC before a final decision is made.

[sdirix]

Related issue: #11924

[msujew]

During the past few years, we've been using npm almost exclusively. It's gotten pretty good and has caught up with yarn v1 pretty well. Yarn was a good decision in the past, but npm is good enough for the project now.

[tsmaeder]

@msujew why not a newer version of yarn?

[guellaeq]

@tsmaeder, I tested the latest version of yarn (v4) and it seems to fix most of the annoying known bugs of yarn v1. It looks like a good alternative to yarn v1. However, there might be a risk when it comes to yarn maintenance. Maybe in the future, new bugs will be discovered in yarn v4 which won't be tackled by the maintainers. Maybe yarn v4 will be abandoned just as the previous versions.
In my opinion, choosing npm is the safest (long term) migration option, since it is by far the most used/maintained package manager in the NodeJS ecosystem.

[tsmaeder]

One point in favor would be the build-in scripting. We have trouble with that when building on Windows.

[tsmaeder]

Also: if we got with yarn 4, we'd have to check whether the dash-licenses tool can read the package.json format.

But whatever we choose, I don't expect the migration to take more than a day or so. Basically, we have to replace all the yarn <script> invocations with npm run <script> and update the documentation.

My vote would be to got with npm because of the lower organizational risk. If we decide it's rubbish, we can quite easily switch again.

[sdirix]

To expand on the original post, these are some of the issues we encountered when working on larger code bases:

• unpredictable hoisting locations: For duplicated dependencies it's often not predictable which version will be hoisted and which versions will be locally expanded. That's an issue in complex code bases with many dependencies
• unnecessary duplication: Although ranges of dependencies are compatible with each other, yarn often does not resolve to a single dependency but multiple ones. Especially for singleton dependencies (i.e. Theia extensions) this is problematic.
• aliasing in resolutions: Aliasing in resolutions (to accomplish transitive aliasing) leads to yarn errors after the yarn.lock was touched once

[dmm9]

Not an expert in build tools, but pnpm is also gaining a lot of traction thanks to faster build times. To the point that it has taken over the lead as the most used monorepo tool according to the State of JavaScript 2023 Survey. Maybe worth having a look.

And also VS Code has switched from yarn to npm:

Beginning of September 2024, we have completed the switch from yarn to npm for package management in the microsoft/vscode repo. This decision was based on the specific requirements of VS Code and center around these criteria:

• Performance: we initially moved to yarn because of performance reasons and npm can now also meet our requirements
• Security: we make our supply chain more secure by limiting exposure and reducing the number of tools we depend upon

[tsmaeder]

Started working on this and it turns out the switch to npm is not simple: npm/cli#3034 Basically, npm does not topologically sort packages in a workspace before running "install" on them. I tend to think this is a killer criterion right there for a project with tens of packages which depend on each other.

[tsmaeder]

VS Code is not really a good example for us as it is not modular like Theia: they do not publish the various parts of their source code as npm modules to be used or left out by adopters.

[tsmaeder]

Basically, npm does not topologically sort packages in a workspace before running "install" on them.

Why this is important: when a theia-package depends on @theia/cli in it's npm install, for example, the @theia/cli package needs to be installed (built) first. Npm does not seem to offer a way to guarantee this. Not even a documented way of determining the order of invocation among workspace packages.

[sdirix]

pnpm is nice and I also use it in some projects. Generally speaking I still favor npm over pnpm for now for Theia because Theia is a large project and npm is the more popular alternative. However we don't necessarily need to leverage npm "Vanilla" but can still use tools like lerna (i.e. nx) which are able to do the topological sorting which we require.

However I can very well imagine that we need to adapt our build system a bit and maybe switch away from a "simple" prepare focused approach. Nevertheless I'm sure that a similar convenient mechanism could be established.

Switching to yarn v4 is potentially less effort (when not leveraging the pnp system), however it's definitely also less popular than npm. My guess is that it's also not without its rough edges and we would be better of with less niche framework. But I don't have any experience with it, so it might be a walk in the park ;)

Overall I don't have hard feelings here and would welcome both, a change to either npm or pnpm.

We could also consider a smaller scale transition first and evaluate using npm in the yeoman Theia generator templates and for the Theia IDE, i.e. for Theia downstream projects.

[JonasHelming]

@msujew IIRC, you wanted to comment here