Isolated workspaces

[bdwain]

@bestander this is based on discussion in yarnpkg/yarn#4099 and would likely address yarnpkg/yarn#4207 and possible yarnpkg/yarn#4219 as well.

rendered view

[bestander]

nit: whole workspace => all workspaces

[bdwain]

oops yea

[bestander]

nit: "Running a workspace" is a bit misleading, word missed?

[bdwain]

Maybe it could say

Working on a workspace in isolation should be as easy as working on all workspaces together.

[bestander]

I think isolated is a better term than regular

[bdwain]

agreed

[bestander]

@bdwain thanks for your RFC!

I was thinking of something way simpler.

Can we have a flag like --isolated that would just install/add/upgrade/ node_modules ignoring that there is a workspace root and other workspaces as well?
Good point on reusing yarn.lock from the root though.

[bdwain]

@bestander yea a flag on the existing commands also works. I kind of mentioned that in the unresolved questions section. I'm not too attached to having it be a separate command versus a flag. I think the general idea is the same though. I was pretty detailed when I wrote it (to cover all edge cases) but I don't think it's a super complicated proposal overall.

If possible though, I would like to retain the ability to run install in a specific workspace without a flag. The reason I think this would be useful is that for a team who is migrating away from a multi-repo setup, people would be used to working on projects in isolation, which means having yarn install behave in an isolated way. For a lot of these projects, it can be the case that the average person working on the project works on a single workspace at a time, and it would be annoying for them to always have to remember to type --isolated whenever running install/upgrade. By allowing them to run the same commands as before, it would make transitioning to a monorepo much simpler, and also just be easier in the long term if you work on isolated workspaces a lot.

I think the simplest way to achieve that would be what I described, where the commands are aware if they are run in a single workspace or at the root of the project. As a slightly more complicated alternative though, there could be 2 flags, --isolated and --workspaces and a default set for the project as to which mode you run in when no flag is set. The default can be in package.json or yarnrc or something, or maybe just a user-specific config value that is not expected to be in source control?

[bdwain]

updated with the wording changes.

[bestander]

BTW, did you know that you can set a flag via .yarnrc for any command?
https://yarnpkg.com/en/docs/yarnrc#toc-cli-arguments

The team that migrates to monorepo can just set

--install.isolated true

In their project's .yarnrc and it will do the trick.

[bestander]

So this is the default mode right now and you propose isolated mode to become default?

I would rather avoid a breaking change and reverse it.
Via .yarnrc you can set up a subfolder to work in isolated mode and if a user cd to that folder Yarn would work in isolated mode.
But if user cd to any other folder Yarn would work the way it works right now

[bdwain]

That works. I did not know about settings flags via yarnrc. I do think it's worth reconsidering the default mode at the next major version, but that can be a separate issue.

So if i used that, i could still go into the workspace and say yarn install --isolate=false to run explicitly in workspaces mode? That is a bit awkward. Kinda why i was thinking of having a --workspaces flag that is basically the opposite of --isolate (in addition to --isolate)

[bestander]

I propose doing a non breaking progressive change first and shipping it.
Then before Yarn v2.0 we can ask people to weigh in if introducing the reverse would be better for them.

[bdwain]

fair enough! I'll make those changes to the doc

[bdwain]

@bestander i changed the proposal to have an --isolated flag

[bestander]

Thanks, @bdwain, I think we achieved a nice balance here.
I am excited to see the PR with this feature, certainly useful for many cases.

[bdwain]

Sweet! I can try to work on a PR to implement this. Thanks for the feedback!

[bdwain]

@bestander I've gotten yarn install --isolated working locally. there's more work to do (other commands and tests and everything), but i just wanted to make sure what I've got so far looks to a good general approach since it's a non-trivial change and I don't have much experience with the yarn codebase. I don't want to get too far in only to find out the approach is all bad.

yarnpkg/yarn@master...bdwain:isolated

basically i just added a rootFolder value to the config object (which takes into account the isolated flag) and use that everywhere instead of using the lockfile folder and assuming it's the installation location.

Do you think you could check out that branch (it's small) and let me know if my general approach looks alright? Or if you're busy if you could point me to someone else who might be able to look?

Thanks!

[choffmeister]

I am looking forward for this to be supported by yarn natively. Until then I wrote a little package that provides a way to isolate a single yarn workspace:

https://github.com/choffmeister/yarn-workspace-isolator

So far it worked pretty well for me (I used it to build individual docker images from my monorepo), but there are for sure some edgecases around the corner.

Maybe this can help others, too, until this RFC is implemented.

[bestander]

@bdwain, looks good, can't wait to see the PR!

[marcinczenko]

Can we have a flag like --isolated that would just install/add/upgrade/ node_modules ignoring that there is a workspace root and other workspaces as well?

Hmm...maybe you already cover that, but I do not see it working for the following case.

Say I am having a monorepo:

packages
    package-1     
    package-2
node_modules
    ...hoisted modules
package.json

Imagine this monorepo is to be open-sourced, but at the same time you start working on a client-app. This client-app is not supposed to land in the monorepo, but it has its own repository. When working on it, you will simultaneously work on packages package-1 and package-2, which are dependencies for the client-app, so you would like to take advantage of the monorepo ecosystem to speed up the development.

Thus, imagine you clone the client-app repo to the packages folder. The .gitignore of the mono repo ignores everything except package-1 and package-2, so anything you do with the client-app will only be visible to its own repository.

Now I want to be able to run yarn install. I would like to get this:

packages
    client-app
        node_module
            package-1 => symlink to package-1
            package-2 => symlink to package-2
            ...all other dependencies are here and not hoisted
    package-1        
    package-2
node_modules
    ...hoisted modules
package.json

So in other words I would like that yarn install behaves as usually with respect to symlinking, but does not hoist any other modules that are listed as dependencies of client-app. In principle, it would be possible to hoist other dependencies from the client-app, but in this specific case we do not want it because it would leave trace on the root yarn.lock. The client-app is here just for the convenience of the development workflow, but it will be removed at some point. We also cannot accept situation in which we make it part of the monorepo repository just temporarily - client-app has a license that does not allow it to be published in a public repository (which is ok for the monorepo).

I would imagine that in the root package.json one could say:

hoisting: {
    excludeWorkspaces: [
        'client-app'
    ],
    /// other hoisting related options
}

Does it make sense, or maybe you think this should be approached in some other, better way?

[bdwain]

my thought is that monorepos aren't really designed to have arbitrary consumer packages added temporarily. if the packages are closely related enough that someone would develop them locally against eachother, it probably makes sense for them to be in the same monorepo. If it's just something like a client-app developer is making a PR to the monorepo and wants to test their changes locally (but does not normally work on it), they can always use yarn link to try it out.

[marcinczenko]

@bdwain ok, I should not mention "temporarily" - the key point in my initial comment is that the client-app has more restricting license than the remaining part of the monorepo and cannot be exposed in the same archive. It may happen that they will stay together for a very long time. Thus, using yarn link is not a solution.

[bdwain]

I'm a bit confused. Is the monorepo aware of client-app? Or does it only exist inside the monorepo when working locally? Is this needed to simplify development?

Git submodules/subtrees/subrepos may be a useful tool for something like that either way. I've heard mixed things about them, but they might be useful for adding arbitrary other repos into the monorepo without committing the actual code.

[marcinczenko]

Yes, it may be a bit confusing, but this is actually very common scenario in our environment (mixed Open Source and not open source). So monorepo is both aware and not aware of the client-app. It is aware in the way that it should respond to all automatic symlinking but at the same time yarn install should not leave traces on the main repo. - git-wise - client-app should not leave any traces on the monorepo's git. This means that submodules/subtrees/subrepo will not work I am afraid because the leave trace in the parent repo (correct if I am wrong wrt to subrepo as I did not really use that one) I crossed this out as git issues are not the key problem - e.g. git subtree will leave traces on the repo, but this is not relevant. Hoisting (of the client-app's dependencies) is the real problem as it makes changes to yarn.lock and this is what we want to avoid (we do want hoisting for other workspace - package-1 and package-2 above). So to summarise - git-wise we can make it work. But we need yarn install not to hoist anything from client-app. For client-app we just want that all its dependencies go to client-app's node_modules + regular symlinking to the packages that are in this monorepo (package-1 and package-2 in my original example).

You mentioned before yarn link - so this would work, but it is quite inconvenient - we like to take advantage of yarn handling this stuff automatically.

[bdwain]

I may be mistaken but it sounds like you kind of want the opposite of an isolated workspace. You want to run client-app inside the monorepo but use its yarn.lock, not the main lockfile, to resolve dependencies, right? And then just to use local copies of package1 and package2 instead of copies installed from npm?

[marcinczenko]

I would not say that it is a total opposite of an isolated workspace (in some way it is), but the rest you got right :).

[marcinczenko]

Yarn is introducing the nohoist option for workspaces: https://yarnpkg.com/blog/2018/02/15/nohoist/. Unfortunately, this will not let you ignore whole workspaces from being hoisted as described above.

[bdwain]

@marcinczenko the proposal ended up changing a bit. Check out the focused workspace folder in the accepted folder. It’s a bit simpler now. I think nohoist should work well with it.