rfc: new npm link command

[iarna]

This was originally discussed here: npm/npm#19421

[1j01]

Missing backtick here (and is this supposed to be npm link & npm unlink together (like the heading below)? or is there something missing for npm link?)

[gilly3]

What you are calling "ephemeral" may be a misnomer. The distinction in that use case seems to be that you are linking to a package in a separate repo, and that it may not be linked for all (or any other) developers. Whether or not it is a temporary state for "trying things out" doesn't matter. I have use cases where links will differ between developers, but the links may exist long-term, even permanently.

I work on several different apps that share libraries. Each app or library is in its own repo. Not all developers on my team work on all apps or libraries, so it would be inappropriate to commit a package-lock dictating links to repos that may not even be cloned on other developers' systems.

Further complicating matters, we prefix all our internal packages with our company initials to avoid naming collisions. (These are published in a private registry.) However, when changing directories in the command line, I find it tedious typing that prefix, so I clone the repos into directories named without the prefix. Other developers do not do this. So, where I might want to link to ../fancylib, my co-worker would link to ../abc-fancylib.

The point is that for many developer workflows, linking to local packages is going to be specific to the developer, and not specific to the package. Thus, committing a package-lock with links will cause headaches.

[JDvorak]

I consider npm link to be a local configuration override on the default behavior, not a specification to be shared between developers.

For instance, if I am working on an ecosystem of modules, then I will be (tirelessly) maintaining links between my local copies in a directory, or between my version of a microcomponent and the surrounding ecosystem depending on it. But, when I incrementally publish my improvements, I would not expect the links to be carried. Nor, would I want them to-- especially in cases where there is a 1-component:1-developer workflow.

[evocateur]

@JDvorak npm publish never includes package-lock.json, nor is it ever respected by a consuming npm install <pkg>, why would it matter what local links exist in it?

[JDvorak]

Because it is respected by anyone running npm install from within a cloned repository.

[JDvorak]

On further reflection, I realize that there are two different understandings of what a dependency is:

1. A dependency is a parameter of a module, and therefore a contingent attribute of its runtime. Like a component in an assembly. "A module is itself only logic overlaying a configuration of other modules, and may over time or in different conditions depend on different modules. Compatible versions, and entirely different modules can and should be swappable." This metaphor arises in the module developer's use of npm link as an explicit alternate binding, and the praise of semver as ideal. You can also see this metaphor in action when someone uses proxyquire for testing, or writes a shim to approximate the separation from the declaration that something is required and the initialization of a compatible instance. I think you might also see this reasoning in most discussions of npm aliases.

2. A dependency is a part of a module, and therefore an essential attribute of its functioning. Like an organ in a body. "A module is an immutable block of code composed of smaller immutable blocks of code. Every call to it should be exactly the same, no matter the conditions or time." It is not a variable element but a specific value. The metaphor of choice for pragmatists who recognize irreducible coupling between modules in business logic. Here I feel is where the idea of package-lock appears obvious, while in the other world it appears equivalent to caching layer.

But, if you notice, the community seems to keep reinventing ways to express a separation between a variable's declaration and its initialization. The npm link I would want is one that allows me to set a different runtime initialization without changing the formal declaration of dependency.

[Dwood15]

@JDvorak

The npm link I would want is one that allows me to set a different runtime initialization without changing the formal declaration of dependency.

I'm having trouble seeing the difference between your version of npm link and the aliasing that was just ratified.

[JDvorak]

If I could just do what I used to do with npm link (and it stops breaking when I run npm install) but under the hood it merely aliases to local directories, well then that sounds great. And, on reading, it does seem that is the case. But, now I'm not sure why we would need a new npm link if we are getting a proper aliasing feature.

[iarna]

@gilly3

What you are calling "ephemeral" may be a misnomer. The distinction in that use case seems to be that you are linking to a package in a separate repo, and that it may not be linked for all (or any other) developers.

By "you" I think you mean @giltayar, who I was quoting there. (Tagging them so they can clarify their intent.) What I understood them to be asking for was very-short-term links, something closer to what npm's current behavior results in (where they're kept until you run npm i again).

The point is that for many developer workflows, linking to local packages is going to be specific to the developer, and not specific to the package. Thus, committing a package-lock with links will cause headaches.

If I said that your concern is that there will be package lock churn between your installation and your co-workers because you have atypical way of cloning your repos and so when they make links they'd stomp on yours and vice versa?

Would your concerns be alayed by storing this in a different file? (I'm not super keen on introducing Yet Another File, but in so far as I can tell, it's what's here, adding something new or remove the npm link command entirely.)

@JDvorak

Because it is respected by anyone running npm install from within a cloned repository.

It would not. npm link create would be needed to replicate your local linking.

If I could just do what I used to do with npm link (and it stops breaking when I run npm install) but under the hood it merely aliases to local directories, well then that sounds great. And, on reading, it does seem that is the case. But, now I'm not sure why we would need a new npm link if we are getting a proper aliasing feature.

I'm confused as to how you think aliasing would actually solve this problem for you? Can you elaborate? Aliases are published as a part of the module metadata. They change the source of the module for everyone that uses them. They are not a local development tool, but a way to ease the use of forks (so you can switch without changing your source code).

[giltayar]

@iarna

What I understood them to be asking for was very-short-term links, something closer to what npm's current behavior results in (where they're kept until you run npm i again).

Yes! With the caveat that removing the links on each npm install/ci/update is a bit too trigger happy: I npm install/update all the time, and don't want my links to be removed when I do. I believe that @gilly3 has a point: the word "ephemeral" is not perfect. Maybe "private": I want "private/ephemeral" links to stay until I manually decide to remove them, but I don't want them officially introduced to my team members, as they are private to my workspace.

Would your concerns be alayed by storing this in a different file? (I'm not super keen on introducing Yet Another File)

I'm really not keen on introducing another file either! You could put the information in the npm home folder (usuually ~/.npm?), keyed on the file path of the package (the package that is linking to another package).

[wearhere]

+1 for @giltayar's "ephemeral" use case as described here. My coworkers and I use links all the time but I'm pretty sure we wouldn't be interested in sharing them with each other. All our packages are in different repos; and so even if multiple developers were working on the same branch of repo A, which branch required that each developer instantiate a link, committing that link to the branch would not be sufficient to "enable" the link for all developers and would probably be otherwise confusing.

I am very interested in not having npm install blow away links though (which I imagine was part of the motivation for saving to the lockfile).

Separately, very interested in having npm unlink undo npm link and restore a non-linked copy of the dependency, which I had filed as https://npm.community/t/npm-unlink-foo-doesnt-reinstall-the-real-foo/713 before being directed here.

[Venryx]

I just want to add a quick note, for others using npm 5 and wanting to restore the old behavior, where "npm install" does not delete symlinked packages: try updating to the latest version of npm 5. (npm 5.10.0, atm)

I had been using the npm 5.6.0 that came with node 8.11, but then saw this comment. I tried updating to the latest 5.X version of npm, and this seems to have restored the old npm install behavior where it just ignores symlinked packages. (or at least doesn't delete them)

This is good enough for me, until we get a revamped npm link system in npm 7. (the npm 5.6.0 behavior of silently deleting all symlinked packages, on any npm install command, was horrendous -- not only did it remove the symlinks, but once it even deleted all the source library folders, which would have meant days of work lost if they weren't also stored on Github)
==========
Oh also, another +1 for @giltayar's "ephemeral" use case. That's how I use linked packages as well, as a way of locally "overriding" a package's contents with a customized version, without having to change anything in the actual project repo. (eg. creating a new web-accessible variant and changing project.json to point to it, or committing the change directly into a now-partially-git-managed node_modules folder)

[thw0rted]

Does anybody else have concerns about the revised link having no support for pack / publish? I currently develop an application and its dependency modules in tandem using link and then distribute the application to our test machines using pack. Today's (6.x) behavior kind of works, but the whole dependency directory is shoved in which is obviously sub-optimal.

I'd love to see pack recurse into linked deps and pack them, then use the resulting output in lieu of a straight directory copy (6.X) or just giving up (this RFC). Does that make sense? Is it too difficult to implement?

[stacktracer]

Anybody know the answer to the following question (copied from the previous thread)?

2. What happens when the version changes in a linked package? When is it noticed? What is the expected behavior if the new version causes it to no longer match the dependency spec of the consuming package?

[giltayar]

The latest commit (npm-link: Revise for non-lock-file storage) looks perfect. Just what the doctor ordered. My monorepo build tool just got simpler! :-)

[zkat]

Yay Ratification!!one

[ties-s]

So what's the status on this? npm 7 still deletes linked modules

[Venryx]

@ties-s What version of npm did you try it with?

I ask because, in this issue, an npm dev (@nlf) suggests the bug may have been fixed by some changes to npm link between version 7.2.0 and 7.5.2, but the OP did not confirm whether those changes fixed it or not. (the lack of response suggested it was resolved, though)

It will be good for someone to confirm if the bug is still present in the latest versions. (7.5.2+) [I'll eventually test it myself if no one else gets to it, but am too busy/lazy atm.]

[ties-s]

@Venryx I have npm@7.6.1

[ljharb]

Try v7.7.5 just in case, and if there’s still problems, please file an issue.

[cyberbiont]

Just quickly tried it with npm 7.8.0 and the bug is still there.

[joe-gre]

I tried with npm 7.8.0 and the bug is still there. Running npm install removes linked packages. Also running npm link <package_name> removes all other links, which is a newer issue that didn't occur on more recent versions of npm whereas the other issue has occurred since npm 5 I believe. The only way to setup multiple links is to run npm link <package_1> <package_1> and run this command again after running npm install.

This is a major issue for my team. We have to waste a lot of time relinking after installing new dependency versions.

[thw0rted]

@joe-gre it sounds like you need to comment on npm/cli#2372 which is marked as closed, and apparently shouldn't be.

[IronGeek]

Still happens with npm 7.11.2... 😞

[baelter]

And 7.18.1 ...

[8enSmith]

And 7.24.0...

[and0p]

I tried with npm 7.8.0 and the bug is still there. Running npm install removes linked packages. Also running npm link <package_name> removes all other links, which is a newer issue that didn't occur on more recent versions of npm whereas the other issue has occurred since npm 5 I believe. The only way to setup multiple links is to run npm link <package_1> <package_1> and run this command again after running npm install.

This is a major issue for my team. We have to waste a lot of time relinking after installing new dependency versions.

This threw me off as well, since an NPM upgrade on one machine made only the last-linked stick around. I now have a link.sh in the repo for all of my projects that need it, which runs npm install followed by npm link local1 local2 .... It's not ideal but saves on a lot of typing.