Injecting workspace packages is a foot gun

[vjpr]

Reference: #3915

I think this is going to be a foot gun. People are going to use it to solve all their inevitable peer dep issues, but then iterative development workflow is completely broken as you would require pnpm i for every change. Any codebase that uses this will be harder to work with.

For every react update, you are going to end up supporting multiple versions, hence everyone will just hard-link to avoid fixing things properly.

Changes

• Feature should not be recommended unless absolutely necessary, and have a warning in the docs.
• Add a .npmrc config option to disable it, or configure it for all packages for testing purposes.

Alternatives

• --experimental-loader hook in ESM to use the correct version
• Tool to help migrate to a single version of a dep across a codebase (which is what the Googles and Metas do IIRC).
• Watcher to add/remove hardlinks when files added/removed

[zkochan]

Add a .npmrc config option to disable it

Why is it needed? "injected" is set explicitly for every dependency that needs it. We did not add a setting to turn it on for all dependencies.

[vjpr]

"injected" is set explicitly for every dependency that needs it. We did not add a setting to turn it on for all dependencies.

I guess this is a good way to dis-incentivize it, by not allowing global configs.

My use case would be: if I am publishing to npm, before I run tests, i would want to hard-link everything to ensure that peer deps resolve correctly during an install. Maybe this feature already exists, I think I remember something like it... This is why I would like a global option.

I would never see myself using the per-package option. And I guess once someone uses it, there is no way to remove it, because the wrong peers might be used.

[defjosiah]

This is semi-related, I had an alternate approach for this same problem:
#3697 (comment)

tl;dr: Use the the explicit "files" property in the package.json to symlink just the files to the proper place (solving the need to re-run an install when files change), and then pull the "dependencies"/"peerDependencies" locally as if it came from the registry (so you get the correct directory structure/peer dependency installation).

If the "files" property changes, you'd need to re-run an install to symlink the new directories/files.

[zkochan]

with hard links a re-run is only needed if new files are added to the package.

[defjosiah]

Oh awesome! I will check this out
This might be exactly what we want 🎉

[genshinw]

I don't know why the hard-linked file not update especially in vscode. I want to know is there another solution for peer dep issue?

[guillaumewuip]

Can we imagine a pnpm install flag like pnpm install --injected to be used when installing in "injected mode" in a specific package of a monorepo? Having a dedicated flag can be cool to provide a good dev experience for the monorepo.

In dev mode, we can install with development dependencies. So no need for injected / hard link workspace: deps nor dependenciesMeta.*.injected in package.json. dependenciesMeta.*.injected can make local dev a bit complicated because of the need to pnpm install again when packages change.

In production mode (eg. in a Dockerfile) with the --injected flag, we can easily install prod deps only and hard link workspace: libs.

// dev mode
pnpm install 

// prod mode
pnpm install --injected --prod 
# only prod deps installed
# all "workspace: " dependencies are hard-linked instead of symlink

I'm maybe missing something here. Let me know if it's something worth discussing / test or if it's better if i move the discussion in a new Discussion :)

[grebenyuksv]

has anyone solved this? I'm thinking of trying to code that watcher which would be re-creating the hard links every time we add files to the hard-linked packages😕

[genshinw]

Do you code the watcher? I need it as well 😂

[grebenyuksv-preply]

We concluded that the watcher would have to be less trivial than calling pnpm i upon file tree modifications.

The problem is that pnpm i seems to be removing and re-creating the hard links to untouched files. Our dev tools weren't able to digest that 🤷‍♀️

[klingenm]

I created a utility package that symlinks the files and folders in the root directory of any workspace package that is using dependenciesMeta.*.injected. I thought this was a good idea, only requiring a new install after adding or removing a file from the root directory.

In my case, I'm using rush to manage the monorepo, keep versions consistent, etc.

This really turned out to be a foot gun. Everything looked as if it worked until running rush deploy ... which simply ignores any symlinks inside node modules package directories.

[iDVB]

Anyone have a solve for this?
The issue we are seeing is similar when it comes to our CI
We use Nx to manage our monorepo.

This is our CI:

1. pnpm install
2. build module1
3. build website (dep on module1 w linkage)

This was failing since module1's dist files get built in step2 after step1's hard linking.
We thought it would be smart to add pnpm install to the build command of module1
Eg.

scripts: {
  "build": "rollup -c && pnpm install"
}

But now this issue when there is a cache hit for the module1 build command it, of course, does NOT run the build command and thus also does not run the pnpm install that is tacked on to the end. 😖

[genshinw]

I have the same problem, do you find any solution to handle this case?

[GiancarlosIO]

has someone found an alternative for the injected feature? 😭

[NullVoxPopuli]

I have this plan/idea, but have not had time to implement: #4965 (comment)

[TimHaboldt]

Is there any update regarding this issue?

After following way too many threads, searching for solutions all over the internet and stumbling over issues regarding the injected keyword with other frameworks I am revisiting the concern once again.

Out Setup
We utilize pnpm as the foundation for our mono repository. It manages all the Node dependencies, and all scripts outside of the Node ecosystem are incorporated into the package.json scripts and can be invoked via pnpm. Our structure differentiates between packages and services. Packages, which contain reusable source code, are used by other packages or services. Services, on the other hand, are deployed into our Kubernetes cluster in the form of deployments, cronjobs, and so forth.

To use the packages in other packages or services, they must first be built since we primarily use TypeScript. Once built, the package can be included via the workspace protocol in the package.json. To enhance the developer experience, we have implemented file watchers for most packages, which can be used in conjunction with hot reloading or file watchers in the services. This accelerates the development process. Changes to a package are synchronized to the node_modules of the service and are detected by the service's file watcher, triggering a rebuild of the service. This efficient setup promotes rapid local development and quick iteration steps.

Moreover, this means that only one version of a package is present at any given time. We currently version very infrequently, and most packages are used directly from the disk, even in production (although the process there is slightly different but not relevant for this discussion).

Understanding the Issue

The workspace protocol of pnpm symlinks packages by default. This approach offers an advantage: any modification to a package is immediately synchronized across all other packages or services using it within the mono repository. This enables swift development workflows wherein packages have their own file watcher, and services can automatically hot-reload their website, incorporating the package changes.

However, a significant issue arises: Peer dependencies are completely disregarded, and due to Node's node_modules resolution, this cannot be easily rectified. The problem occurs because Node follows symlinks to their actual location on the disk, which is the folder in your mono repository where your package resides. From there, it reads the source files of your package and starts resolving package imports. These imports are resolved from the context of your mono repository package which in this case is the folder where the package is located. This becomes problematic with peer dependencies.

Peer dependencies are used to inform a package that it needs to have a specific package installed to function, but it doesn't necessarily care about the version of the package. This is crucial for packages like React, where only one React version should be used per webside due to global singletons, etc.

This means that if service A has React 16 installed and another one has React 18 installed, the workspace package using React needs to select a different React version depending on where it's being used. However, due to how Node resolves the symlinks, you only have one package location to work with, and installing both React versions in one node_modules won't work.

To address this, pnpm introduced the dependenciesMeta.inject option, which allows hard-linking packages into the .pnpm folder instead of symlinking them. This generally resolves the issue, as hard-linked packages will not be resolved by Node to their source. This allows multiple copies of the hard-linked package in the .pnpm folder, each using different React versions in their node_modules. Now, packages can have peer dependencies, and the local version is used.

However, this solution presents a new problem: synchronization of file changes.

The Problem with Hard Links

Hard links only reference the content of another file. If you're familiar with C++, a hard link is essentially a pointer. It points to the contents of a file. If any pointer to the contents of a file exists, the content will still exist. If all pointers are deleted, the file content will also be deleted. In our case, it means that if you run, for instance, tsc to recompile the dist folder and it recreates the files, the hard links are essentially lost. The files in .pnpm still point to the old file content. Moreover, new files and deleted files are not detected at all. This is why you need to run pnpm install after each package change.

This is very inefficient and slow to work with.

Moving Forward

I understand that this issue is not rooted in pnpm and cannot be easily fixed by pnpm. However, I'm curious to know if there are any ideas on how to proceed here, @zkochan. Has anyone created any issues in the Node GitHub repository? Are there any planned updates that might alleviate the pain points with hard links?

This issue is frustrating because it means that pnpm doesn't work with one of the core features of the Node ecosystem. Peer dependencies are used everywhere, and if you want to use packages without publishing them to a registry, you can't avoid them at all. This is evident from the number of issues created regarding peer dependencies. Most of them are completely confused on why this is the matter.

Because other solutions are not very appealing. For example, Nx recommends installing all dependencies of a mono repository in the root package.json of the mono repository source. We strongly disagree with this approach because it means that every package you want to update requires you to update the entire mono repository. This becomes extremely complex and challenging, especially when packages are using all over the mono repository and changes introduce complete overhauls in their usage.

Solving this issue by not building the package itself and instead building it from the service isn't viable either. First, deletions and file creations are not picked up but can be fixed by doing pnpm install. But we also noticed that file watchers of webpack or nextjs had a lot of troubles picking up the file changes if the content of a hard-linked file got changed. Most of the time, they didn't rebuild or just completed in an unfinished state, which makes our use of hard links practically impossible.

There are even more examples of where this structure is not working. For Turbo, the Nx alternative, it allows you to define pipelines. For instance, if you want to build a service, you can define a pipeline that builds all packages required by the service before building the service itself. This process is similar to running a pnpm --filter service-a... build command. It also adds smart caching to the mix.

However, this entire process falls apart because after a package has been built, pnpm install should be executed. Upon conducting some research, it appears that there's no viable solution to this issue in Turbo.

Conclusion

Can anyone tell me if there are any solutions out there that make this whole matter easier? Are we using pnpm in combiation with the mono repositroy wrong?
Did anyone else solve this issue while maintaining the fastest development speed possible?

I invite further comments and suggestions on this topic, as I find it crucial for the pnpm community to find a way to make peer dependencies work.

[NullVoxPopuli]

direct link to that project: https://github.com/NullVoxPopuli/pnpm-sync-dependencies-meta-injected

[defjosiah]

Thanks! I think I meant to paste that link 😂

[TimHaboldt]

I've actually seen the package before, popping up in some threads. However, we haven't finished evaluating it internally yet. I'm not sure why I forgot to mention it in the previous post. Thanks for providing the link anyway 😄.

There are two main reasons why we haven't started using the package yet.

Firstly, it introduces another layer of knowledge that engineers in our company are supposed to have in order to use the mono repository properly. Not only do they need to know that the workspace protocol breaks for peer dependencies, which is already problematic by itself, but they are also required to add another layer on top by implementing the script inside the package.json and referencing it during the start command. It isn't a bad solution, don't get me wrong. I'm really happy that @NullVoxPopuli found a valid solution. But honestly, introducing more complexity to a setup that already seems quite complex is not really desired by us. We might still use it because there aren't any other solutions around though.

The second reason lies in how we want to manage the local services in the future. We are probably going to introduce skaffold soon, and from what I've experienced personally, Docker and the permissions used inside the container will most likely affect local files. Having the sync package active might affect the .pnpm folder, which will break pnpm locally. I'm not sure if this issue will be the case and it can most likely be prevented (that's why we are still in the evaluation phase of the package 😉). But it is adding yet another layer of complexity.

I just wish pnpm would introduce something like the package from @NullVoxPopuli, especially because it has been around for a while. If we could natively introduce its functionality into pnpm, it would already eliminate most pain points. @defjosiah's approach of providing a workspace watcher doesn't sound too bad either.

[TimHaboldt]

There is also the option of making the developers of node aware of this issue. If they for example provide a way of telling the package resolver from which location they should start resolving packages for symlinked files it would probably already be enough to make this whole Issue go away.
A check for: if you ever end up in a subfolder of .pnpm. Start resolving packages from this point onward or smth.

[Ugzuzg]

Another issue I have discovered is that the resolution of the dependencies of an injected package doesn't happen or doesn't work correctly. Ideally, it should treat every dependency as if it was installed from a registry.

Turned out to be an issue with the naive approach that pnpm-sync-dependencies-meta-injected takes, not with pnpm. With my dependency tree, the monorepo packages are not only injected into the packages in the repo itself, but also into node_modules/.pnpm store, which pnpm-sync-dependencies-meta-injected doesn't take into account. If I use pnpm install after every single package is built, then the whole thing seems to work.

It's a hard problem to solve in a paradigm of "pnpm is a package manager". It would be easier if the paradigm was "pnpm is a build system".