"yarn knit": a better "yarn link"

[ide]

This is a feature request to make "yarn link" or a new command be more useful than the current behavior, which was modeled after "npm link". Before focusing on the actual linking mechanism I want to describe the desired behavior of "yarn link":

Desired behavior

The "yarn link" workflow should mimic publishing a package (ex: dep) to npm and then installing it in a dependent (ex: app), and keep this constraint while you're making changes to the first package. Concretely, "yarn link" should make it so that when you save a change to dep, the resulting state is as if you:

1. Ran "npm publish" in dep (assume that it can clobber an existing version, and that you're publishing to a local registry on just your computer).
2. Ran "yarn add dep" in app.

Why this behavior is great

This solves several problems that "yarn link" has today:

Isolating node_modules correctly

You can install dep in two different apps without sharing the node_modules of dep. This is a problem with Electron apps, whose V8 version is different than Node's and uses a different ABI. If you have node-app and electron-app that both depend on dep, the native dependencies of dep need to be recompiled separately for each app; node-app/n_m/dep/n_m must not be the same as electron-app/n_m/dep/n_m.

Working on multiple versions

You can be developing multiple different versions of dep. Say you have two directories, dep-1 and dep-2, which have your v1 and v2 branches checked out, respectively. With "yarn link" it's not possible to make both of these directories linkable at the same time.

This is a problem when you are developing & testing dep-1 with old-app and dep-2 with new-app. You don't want to be going back and forth between dep-1 and dep-2 running "yarn link" each time you switch which app you're testing.

Faithfully representing the node_modules hierarchy

Currently "yarn link" symlinks the entire package directory, which brings along its node_modules subdirectory with it. With dependency deduping and flattening, bringing in dep/node_modules wholesale usually produces a different node_modules hierarchy than running yarn install in app and installing everything from npm. This isn't a problem most of the time but it does go against Yarn's spirit of consistency and the lockfile.

A practical proposal -- knitting :3

This is a proposal that solves all of the problems above and isn't too hard to implement or understand. I'm going to call it "yarn knit" to distinguish it from "yarn link". Conceptually, we find all the files we'd normally publish to npm, pack them up using symlinks instead of copies of the files, publish the pack to a local registry (just a directory), and then when installing we look up packages in the local registry directory instead of npm.

Running "yarn knit" inside of dep

This is the step that simulates publishing dep. Running "yarn knit" in dep finds all the files that "yarn publish" would pack up and upload to npm. Crucially, this excludes node_modules, and would follow the same algorithm as "yarn publish" such as reading package.json's files field.

Then it simulates publishing dep: it creates a directory named dep-X.Y.Z (where X.Y.Z is the version of dep in its package.json) inside of a global directory like ~/.yarn-knit. A symlink is created for each file or directory that "yarn publish" would normally have packed up. This step shares some conceptual similarities with publishing to a registry, except it uses symlinks and it's local on your computer.

Running "yarn knit dep" inside of app

This behaves like "yarn add dep" except that it looks at the versions of dep that are in the global ~/.yarn-knit folder and takes the latest one. (You also could run "yarn link dep@X.Y.Z" if you wanted a more specific version, like "yarn add".)

"yarn knit dep" then runs most of the same installation steps that yarn add dep would. It creates app/node_modules/dep and creates symlinks for each of the symlinks under ~/.yarn-knit/dep-X.Y.Z. Then it installs the dependencies of dep as usual by fetching them from npm. Finally it runs postinstall scripts.

Issues

One issue with this proposal is that it's not clear what to put in the lockfile after running yarn link dep since we don't have an npm URL for the dep yet -- it hasn't been published to npm. Another issue is that if you change package.json in dep, namely changing a dependency or modifying the files entry, you have to run cd dep; yarn knit; cd app; yarn knit dep.

[ide]

(Edit: this is in response to a comment that pointed out another issue I forgot to mention. If you update the code in dep and bump its version, say from 1.0.0 to 1.1.0, the symlinks in ~/.yarn-knit/dep-1.0.0 will still point to the code in your working directory, which now contains 1.1.0 code.)

The symlinks might break but I think that's mostly OK since at that point you're done working on dep and have published it to npm and it's easy to go run yarn add dep in app and not use the symlinks anymore.

If you want to truly pin the versions of knitted packages then you'd need to have a different working directory for each version. (Git worktrees are great for this use case actually. Worktrees let you check out a repo once and then magically create semi-clones of it in separate directories, with the constraint that the worktrees need to be on different branches, which is totally OK in this scenario. The worktrees all share the same Git repo though, so if you commit in one worktree you can cherry pick that commit within another worktree.)

[sebmck]

This is a substantial feature request so please open this on the yarnpkg/rfcs repo.

[wolfgang42]

I've copied this issue into yarnpkg/rfcs#30.

Edit, July 2017: The above issue was closed abruptly and subsequently deleted; the proposal has now moved to https://github.com/yarnpkg/rfcs/blob/master/accepted/0000-yarn-knit.md

[nrser]

@ide we really want/need this... we are in the process of developing out a platform, and our preferred approach of using many small focused packages is absolute hell when you are co-developing them.

i wrote an ansible playbook to try and handle "knitting" the local packages (which are git submodules), but it's half-baked at best; we would love to have a solid solution incorporated into yarn, and would be happy to contribute to building out the feature.

i read the RFC... it's not clear to me where to comment on (hence reply here) it or what's going on with the process?

[ide]

I'm not sure if anyone is working on this (@bestander maybe?). If not you might be able to coordinate with the Yarn team and talk about what a PR would look like.

[nrser]

cool, thanks.

looks like you've done some contributing; what's the best way to talk to the yarn team? https://discord.gg/yarnpkg ?

as a side note, playing around with yarn 0.21.2 and looks like it now symlinks directly to the destination instead of going through a global directory, which would seem to mean you can use it link up packages and dependencies without worrying about a later yarn link changing where they point... though i've just started testing this, so not yet sure.

[bestander]

Discord https://discord.gg/yarnpkg could be a good place for group discussions with the core team.

We are currently working on "workspaces" feature that should address workflows of multiple packages in a single source repository.
As for this RFC it is up for grabs, I don't think anyone is planning to work on this.

[MikeTaylor]

@nrser wrote:

we really want/need this... we are in the process of developing out a platform, and our preferred approach of using many small focused packages is absolute hell when you are co-developing them.

May I add a big AMEN? This is precisely how we're working too, and while yarn link is a big step forward from npm link, it's still far from the right thing. I would love to see yarn knit implemented.

[nrser]

@MikeTaylor the (i think more recent?) yarn link behavior has been enough for me for the moment... i'm using this script to tie the dependent sub-packages together:

https://gist.github.com/nrser/9bc033cc59fa59f5ef3f2b5242892736

it's literally a few dozen lines but it band-aided my situation enough to keep going. it doesn't handle peer dependencies and will probably loop forever if you have circular deps among i'm sure a host of other issues i have yet to run into, but wanted to offer it up as a quick fix / starting point if you like.

[nrser]

thanks @bestander. i've managed to get my stuff working with a few dozen lines of ruby and am going to have to live with that for the moment... just can't spend the cycles on it right now. i'll let you guys know if i find some time to take a crack at it.

[MikeTaylor]

Thanks, @nrser. It's a bit sellotape-and-string, but better than nothing. But how I wish that yarn link would Just Work.

[wclr]

@ide

Then it simulates publishing dep: it creates a directory named dep-X.Y.Z (where X.Y.Z is the version of dep in its package.json) inside of a global directory like ~/.yarn-knit. A symlink is created for each file or directory that "yarn publish" would normally have packed up. This step shares some conceptual similarities with publishing to a registry, except it uses symlinks and it's local on your computer.

But individual file symlinks still will be pointing to an original location that means that dependencies that are required in those files will be looked up relative to a location on a disk. Though node actually has workaround for this --preserve-symlinks flag, though it is not working option in some cases.

[wclr]

To address this issue and generally problem of yarn/npm link workflow I made simple local package manager https://github.com/whitecolor/yalc it implements knitting, though not sure if it is really useful )

[wolfgang42]

@dandv I've edited my comment to explain that the issue I created has been deleted, and to link to the RFC. Thanks for tracking this down; there seems to have been some early changes in the Yarn RFC process that weren't communicated well.

[dandv]

@wolfgang42: Thanks. Deleted my previous comment since it's obsolete now, and will delete this one too.

[silouanwright]

@whitecolor that yalc package works absolutely flawlessly for me. Definitely recommend for everyone to try it out.

[bichotll]

To address this issue and generally problem of yarn/npm link workflow I made simple local package manager https://github.com/whitecolor/yalc it implements knitting, though not sure if it is really useful )

Definitely, the way to go for now. I hope yarn or npm ends up implementing something similar because linking React is such a headache.