-
Notifications
You must be signed in to change notification settings - Fork 596
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
build tools: Yarn 2, yarn 3 and NPM 7+ for Monorepo Solution #4737
Comments
From my own grueling experience over the last three weeks, yarn 2 has some issues with running scripts across all packages in a monorepo, and lerna still seems better suited for running scripts. I am using yarn 2 workspaces with pnp and their no install methodology and it works great. It is just running scripts that has been my biggest issue. I'm sure it is something that can be worked out with the right package config, I just haven't found it yet. |
Thanks @KingOfTac for your insight. |
@Ibrahimmaga thanks for the detailed analysis, could you please expand on the following for each option (NPM, Lerna, Node, Yarn)
Thanks, |
I just got finished moving from yarn1 to npm7 there were some hiccups but thankfully it's all looking good now. I even got to drop npm7 + changesets is looking pretty good if you can pull it off |
I'm still having issues with typescript not finding package imports when using yarn2 with pnp. I haven't tried npm7 yet to see if it is any better. |
|
@awentzel thanks for summarizing these. Are there any notable criticisms out in the wild of npm 7? It looks like it addresses the major issues that I've encountered with npm. Tossing |
@Ibrahimmaga how would these commands change and how does NPM 7 process the equivalent logic from our publishing pipeline here? |
We will follow this [pipeline template here](https://github.com/microsoft/fast/actions/runs/1066482693/workflow) in terms of configuration. The rest will be npm 7 commands. The publishing will be beachball still.
|
I think for the new tooling repository and the creator repository these can both be npm 7 with custom build scripts for anything extra. I would like to see if we can skirt around using Yarn 2 and Lerna if possible, and it seems like this is do-able with the current npm 7 feature set. |
I think I missed it - but is there a reason Lerna is still required with Yarn 2? Yarn includes workspaces as well and I’m not sure we have an accurate comparison if we’re keeping Lerna in the picture when it’s unnecessary from my understanding. Cursory read of above, so perhaps I missed it :) editing to include a link: https://yarnpkg.com/features/workspaces |
Lerna is not required with Yarn 2. but for a large project, it is nice to have the bootstrapping of the build and get a visual representation that Lerna provides. Otherwise, Yarn 2 and NPM 7 can handle workspaces and crosses dependencies. So I'm sure we don't need Lerna the creator project. |
Thanks @Ibrahimmaga - I definitely need to read up a bit more on the above. I’m not speaking in terms of creator, but this entire monorepository. I see a lot of references to Yarn 2 + Lerna but I think that’s presenting a bit of a slant towards NPM, mostly due to the fact that the intent here is trying to move away from Lerna so that’s not enticing - but as you’ve said it’s not accurate. Curious if we have info comparing workspaces in both and if we’ve dug into the node_modules “replacement” text - I just read through after seeing @nicholasrice’s concerns and can appreciate some of their thinking. I’m not committed to one or the other, but I do want to make sure that we’ve done our due diligence on what matters to a project of our size. Converting is costly and making the right decision here can save headache and headache for all of us. Thanks for digging into this for us and sharing the findings! |
Testing only changed packages could be a good idea for performance. Sometimes the changes affect another package and can go without notice. I think it is better to trade in performance than fixing breaking changes that could take hours to fix. The ideal solution is to test the whole application if there is any change. |
I think it's a good point though, there are several tools being used currently that look up the mono repositories package dependency tree, including beachball. If you can know what the dependents are on a changed package you can only run the tests for the affected packages. I'm going to make a note of this for the migration work for fast-tooling and see if we can't find a good solution because the performance boost would be nice. |
If this happens, it probably means there are bigger problems related to implicit dependencies not listed in |
By using a workspace option, we can use npm command that will run a specific test script within the packages. |
@Ibrahimmaga please spend some time addressing Chris's concerns here clearly articulating how each handles these scenarios.
Thanks @chrisdholt for summarizing your concerns. This is great feedback and Ibrahim will continue to investigate. I think up to this point we were leaning on NPM7 only. Yarn 2 seems to be unstable still for some teams. I think we're all in agreement that a key requirement is reducing dependencies as much as possible. Perhaps Ibrahim can do more research here as well for Enterprise projects. We should be focused on comparing similar projects (apples to apples) and researching to this capacity. |
What template @Ibrahimmaga ? |
Our current pipeline template: (https://github.com/microsoft/fast/actions/runs/1066482693/workflow) someone has asked question if this will changed. |
Here a Yarn and NPM comparison side by side by TechGenyz |
From the peanut gallery, we have been developing a design system monorepo using fast-foundation and fast-element. Our packages are roughly: components, tokens via amazon style dictionary + Adobe DSP, angular wrappers, and sample apps for vanilla applications and angular applications. So far we have not had any issues with npm 7 workspaces. The only speed bump so far is the lack of microsoft beachball support for managing the npm package-lock.json file: microsoft/beachball#525 |
Thanks for your response @rajsite. I used beachball with yarn, there maybe a new configuration for beachball since the release of NPM 7. You should contact kchau@microsoft.com or @ecraig12345 Elizabeth Creg for beachball support. |
Due to time constraints I mostly only work on beachball as features are needed for my team, unfortunately (same with @kenotron for the most part). But I can help with PR review if someone else wants to implement this. |
I'm done with this research, but I want to reply to this question. when using Yarn, Npm has to be installed first, Some may claim, NPM and Yarn are 2 differents mono repo solutions, but I see Yarn as an extension for NPM since it is not a standalone application. In the background, Yarn is using NPM to manage node module and have added functionalities that NPM failed short. Without NPM installed, you won't be able to use Yarn. |
@Ibrahimmaga Could you elaborate on that statement? Yarn is literally a single JS file so it doesn't need nor use npm to function, it's true we recommend installing it though
@awentzel Could you open some issues for the "unstableness" you've encountered? It's difficult to improve if you don't know what you need to improve
All stable versions of Yarn has
@KingOfTac Could you open an issue for these problems you ran into? https://github.com/yarnpkg/berry |
There are two feature requests in Yarn that could enable its usage as a Lerna replacement for our team:
Both issues have comments with workarounds but IMO having built-in support is a must in terms of stability. |
I think the bigger picture is that Yarn (2+) lets you easily implement whatever behaviour you need without waiting for the package manager to agree with whatever direction you want to follow, while still benefitting from our core APIs. Here it's changed workspace, but it can also be release workflow, package validation, etc. |
We are continuing to evaluate this. We are trying out a few different scenarios to understand all implications and based on the community feedback and our own conclusions we'll be doing our best to move in the most efficient and productive direction. Thanks, everyone for your feedback on this issue. |
Very nice thread here. I just got done (mostly) implementing Yarn v3 for Nrwl Nx -- see feat(core): implement Yarn v3 Also wanted to share that using @yarnpkg/plugin-patch was insanely awesome and helpful in debugging of issues. Worth looking into! |
Perform an investigation to verify if Yarn 2 and yarn 3 or NPM7, can meet our mono-repository needs as features could have changed since the last time we investigated and we should understand the implications of choosing one of the other and if we can achieve our usual development process without Lerna.
Yarn 2
Version 2 of the popular JavaScript package manager Yarn2 has been here for a year now. Yet try to find information on upgrading from 1.X or how it compares to current versions of npm (especially the 7.x dev release), and you’ll get plenty of well-written but outdated blog posts mainly bemoaning the new direction the Yarn team has taken. On its initial release, taking a wait-and-see approach with Yarn 2 may have been the correct call, but it’s time to reconsider that decision.
Here are three reasons you might have waited to make the switch — and why those reasons are out of date in 2021
Yarn 2 uses a new Plug’n’Play (PnP) architecture that is a huge departure from how npm projects have always worked. PnP does away with the node_modules/ folder where packages get installed. Unfortunately, this is incompatible with certain JavaScript-based projects that rely on that directory to find packages, namely React Native, Flow, and VSCode Extension packages.
However, Yarn 2 now offers an option that copies packages to the node_modules/ folder just like Yarn 1, providing backward compatibility for these projects. It literally requires adding a single line to your new .yarnrc.yml configuration file to turn on this feature.
As the Yarn 2 documentation mentions:
Even if you don’t use Plug’n’Play nor plan to use it, your project will still benefit from more stable node_modules layouts, improved performances, improved user experience, active development, etc.
Sooner or later, you still have to migrate.
Yarn 2 migration and documentation: Migration | Yarn - Package Manager
Step by step migration to Yarn 2
Yarn 2 Documentation for Migration
npm install -g yarn
to update the global yarn version to latest v1yarn set version berry
to enable v2 (cf Install for more details).npmrc or .yarnrc
, you'll need to turn them into the new format (see also 1, 2).yarnrc.yml
fileyarn install
to migrate the lockfileSome optional features are available via external plugins:
Run yarn plugin import interactive-tools if you want upgrade-interactive
Run yarn plugin list to see what other official plugins exist and might be useful
Commit the yarn plugins
Yarn 3.0.0 Release
Breaking Changes- Node 10 isn't supported anymore.
- Plugins can't access yup anymore (we migrated to Typanion as part of Clipanion v3).
- To upgrade workspace-tools, remove it from your .yarnrc.yml, upgrade, then import it back.
- The enableImmutableInstalls will now default to true on CI (we still recommend to explicitly use --immutable on the CLI).
- You can re-allow mutations by adding YARN_ENABLE_IMMUTABLE_INSTALLS=false in your environment variables.
- The initVersion and initLicense configuration options have been removed. initFields should be used instead.
- The -a alias flag of yarn workspaces foreach got removed; use -A,--all instead, which is strictly the same.
- The old PnPify SDK folder (.vscode/pnpify) won't be cleaned up anymore.
- The --skip-builds flag from yarn install got renamed into --mode=skip-build.
- The bstatePath configuration option has been removed. The build state (.yarn/build-state.yml) has been moved into the install state (.yarn/install-state.gz)
- The cache files need to be regenerated. We had to change their timestamps in order to account for a flaw in the zip spec that was causing problems with some third-party tools.
- @yarnpkg/pnpify has been refactored into 3 packages:
- @yarnpkg/sdks now contains the Editor SDKs
- @yarnpkg/pnpify now contains the PnPify CLI compatibility tool that creates in-memory node_modules
- @yarnpkg/nm now contains the node_modules tree builder and hoister
- @yarnpkg/plugin-node-modules has been renamed to @yarnpkg/plugin-nm
- The --clipanion=definitions commands supported by our CLIs will now expose the definitions on the entry point (rather than on .command)
6.Yarn will now generate .pnp.cjs files (instead of .pnp.js) when using PnP, regardless of what the type field inside the manifest is set to.
7.The virtual folder (used to disambiguate peer dependencies) got renamed from $$virtual into virtual.
NPM 7
With version 7 of npm they've reduced their dependencies by roughly 54%, while increasing the code test coverage by about 17%. It should also include a performance boost in multiple areas according to their own benchmarks.
Npm 7 is now the latest version in the npm registry and therefore default. To install the new version of npm, you'll run the following in your command line interpreter of choice:
npm install --global npm@latest
Couple new features from NPM 7
With the new package-lock.json file we'll unlock the ability to do deterministically reproducible builds. It should now include everything npm needs to install the packages needed. Before npm 7 yarn.lock was ignored by npm, but this is no longer the case. It can now use it to keep itself up to date with the package tree.
The new lock file should be backward compatible with users of npm 6. Though, when you run npm install in a project with a version 1 lock file it will replace that file with the new structure. This can be avoided by running npm install --no-save when installing.
This is one of the new features that I'm most excited about. It includes a set of features that will make the management of multiple packages a lot better. It lets you handle packages from a singular top-level root package. This has already been possible to do with for example yarn, Lerna, or Pnpm.
In order to make npm aware that the current project is a workspace you have to add workspaces key in your package.json. This can be done by adding every single sub-folder or by using a glob, like in the example below.
More on workspace: rfcs/0026-workspaces.md at latest ,
more on workspace on npm(workspaces | npm Docs )
Automatically installing peer dependencies
Breaking changes
Since the new version is considered a major version it'll come with a couple of breaking changes. Here are some:
You can no longer use require() npm's internal modules. Npm now uses the package. exports field.
The team has completely rewritten npx to internally use npm exec, the npx CLI will still be available. Some functionality changes are to expect. One is that you'll now be prompted if you try to run a module that is not installed yet.
The changes mentioned above regarding peer dependencies might disturb some workflows.
npm audit has a new output.. npm 6 showed all packages by default when running npm ls. With npm 7 it will only show the top-level packages. Run npm ls --all to mimic the behavior from npm 6.
Node Version
**Node Version 14 recommended**
There are many things to look forward to with this major release. Some of the new features include:
Node version 16 current
Here are some of the new features and ongoing work in the 16 release, which include:
Summary
Before npm7, it was not possible to handle a large monorepo project with npm alone. That why developers switched to yarn. With NPM7, this is no longer an issue, NPM can do exactly what yarn does. npm v7 arrives with a newer version for the package-lock.json format - allowing to reduce the need to read package.json files and to have enough information to reliably describes the full and precise package tree all by itself. More than that, the resulting package tree using the new lockfile is flattened, and this is crucial to boost the performance.
Support. We have a partnership with JavaScript Cloud Advocates v-Team at Microsoft, who provide information on what happening on NPM 7, Node Js, and GitHub. If we start using npm 7, we will be getting support from them.
As GitHub owns NPM 7, the most recent information is posted on GitHub.
npm 7 is now generally available! | The GitHub Blog
Yarn has been a leading Monorepo solution sometimes in combination with Lerna. The early release of Yarn 2 has some incompatibility with Certains JavaScript library like react. Sounds they have provided some fix for it. Here is the official link to Yarn: 2 - Installation | Yarn - Package Manager
The community of Yarn has stated that Yarn 2 is a little slower than Yarn 1. This could be due to changes in Yarn infrastructure and added functionalities. The release of Yarn 3.0.0 has taken the performance into consideration and reduce configuration during the setup process or migration from the old version of Yarn.
Node.js. We will use the recommended version 14. This is to prevent a breaking change in the customer application. If we use the latest version 16. Most customers may not be up to date with their node version.
npm list -g
yarn licenses list
command will list, in alphabetical order all of the packages that were installed by yarn or yarn install, and give you the license (and URL to the source code) associated with each package.npm install license-checker
,npx license-checker
This will give you a printout of all the licensing details of packages used in your project. Other cool features of the project: Print a summary of licenses used bynpx license-checker --summary
. This can also be Include in your CI/CD pipeline by providing it with a whitelist or a blacklist of licensesThe text was updated successfully, but these errors were encountered: