yarn.lock gets updated by collapsing dependencies when there is no need to

[ddgenome]

Do you want to request a feature or report a bug?

bug

What is the current behavior?

When running yarn (install), the yarn.lock file gets rearranged when there is no need to.

If the current behavior is a bug, please provide the steps to reproduce.

I have found the following package.json to create different yarn.lock files on different machines.

{
  "devDependencies": {
    "@types/mocha": "^2.2.40",
    "@types/power-assert": "^1.4.29",
    "espower-typescript": "^8.0.0",
    "mocha": "^3.2.0",
    "power-assert": "^1.4.2",
    "tslint": "^5.0.0",
    "typescript": "2.3.2",
    "yarn": "^0.23.4"
  }
}

Since yarn's heuristics around collapsing duplicates seems dependent on your current local cache, it is somewhat difficult to reproduce. The best way I have found is:

1. Run yarn on the above package.json, generating a yarn.lock, $ yarn
2. Add the generated yarn.lock to source control to more easily detect changes
3. Delete or remove your local yarn cache, $ mv "$(yarn cache dir)" "$(yarn cache dir)-save"
4. Delete the node_modules directory generated in the first step, $ \rm -rf node_modules
5. Run yarn again and the yarn.lock will have changed, $ yarn && git diff

You can find an example here: https://travis-ci.org/atomist/travis-rugs/builds/228523519#L382
The line linked to is the invocation of yarn --frozen-lockfile (the fact that the --frozen-lockfile argument is ignored is discussed here #3313 ) and subsequently git diff is run. You can see from the diff output that the only changes are collapsing of duplicate versions into a single version that satisfies all the constraints.

diff --git a/.atomist/yarn.lock b/.atomist/yarn.lock
index 613e5cf..2a4b6cc 100644
--- a/.atomist/yarn.lock
+++ b/.atomist/yarn.lock
@@ -315,18 +315,12 @@ death@^1.0.0:
   version "1.1.0"
   resolved "https://registry.yarnpkg.com/death/-/death-1.1.0.tgz#01aa9c401edd92750514470b8266390c66c67318"
 
-debug@2.6.0:
+debug@2.6.0, debug@^2.2.0:
   version "2.6.0"
   resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.0.tgz#bc596bcabe7617f11d9fa15361eded5608b8499b"
   dependencies:
     ms "0.7.2"
 
-debug@^2.2.0:
-  version "2.6.6"
-  resolved "https://registry.yarnpkg.com/debug/-/debug-2.6.6.tgz#a9fa6fbe9ca43cf1e79f73b75c0189cbb7d6db5a"
-  dependencies:
-    ms "0.7.3"
-
 deep-equal@^1.0.0:
   version "1.0.1"
   resolved "https://registry.yarnpkg.com/deep-equal/-/deep-equal-1.0.1.tgz#f5d260292b660e084eff4cdbc9f08ad3247448b5"
@@ -962,14 +956,10 @@ mimic-fn@^1.0.0:
   dependencies:
     brace-expansion "^1.0.0"
 
-minimist@0.0.8:
+minimist@0.0.8, minimist@~0.0.1:
   version "0.0.8"
   resolved "https://registry.yarnpkg.com/minimist/-/minimist-0.0.8.tgz#857fcabfc3397d2625b8228262e86aa7a011b05d"
 
-minimist@~0.0.1:
-  version "0.0.10"
-  resolved "https://registry.yarnpkg.com/minimist/-/minimist-0.0.10.tgz#de3f98543dbf96082be48ad1a0c7cda836301dcf"
-
 mkdirp@0.5.1, "mkdirp@>=0.5 0", mkdirp@^0.5.0, mkdirp@^0.5.1, mkdirp@~0.5.0:
   version "0.5.1"
   resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-0.5.1.tgz#30057438eac6cf7f8c4767f38648d6697d75c903"
@@ -996,10 +986,6 @@ ms@0.7.2:
   version "0.7.2"
   resolved "https://registry.yarnpkg.com/ms/-/ms-0.7.2.tgz#ae25cf2512b3885a1d95d7f037868d8431124765"
 
-ms@0.7.3:
-  version "0.7.3"
-  resolved "https://registry.yarnpkg.com/ms/-/ms-0.7.3.tgz#708155a5e44e33f5fd0fc53e81d0d40a91be1fff"
-
 multi-stage-sourcemap@^0.2.1:
   version "0.2.1"
   resolved "https://registry.yarnpkg.com/multi-stage-sourcemap/-/multi-stage-sourcemap-0.2.1.tgz#b09fc8586eaa17f81d575c4ad02e0f7a3f6b1105"

This leads me to think the problem has arisen recently with the work around collapsing duplicate dependencies, see #579 . Thus, this issue seems different than #570, #1576, and #2023 (which turned out to be a non-issue), aside from the fact that those issues are closed and this problem occurs with the latest version of yarn.

What is the expected behavior?

Per #570 (comment) , if the current yarn.lock file satisfies the package.json file, I expect yarn to install the packages according to the current contents of the yarn.lock, not update and rewrite yarn.lock with an equivalent alternative.

Please mention your node.js, yarn and operating system version.

$ node --version
v7.9.0
$ yarn --version
0.23.4
$ uname -a
Darwin mbp.local 15.6.0 Darwin Kernel Version 15.6.0: Fri Feb 17 10:21:18 PST 2017; root:xnu-3248.60.11.4.1~1/RELEASE_X86_64 x86_64

[mysterycommand]

I think I am running into this same issue. It def. seems weird that it's shuffling dependencies when the resultant dependency tree is identical, but I think this comment might also apply here:
#3313 (comment)

@kittens, @thejameskyle, sorry for digging up an oldy, but is what you meant by this comment:
#570 (comment)

… that a package.json created by adding dependencies with the --exact flag should never result in an updated yarn.lock on yarn install? Because I think I'm seeing dependency differences across machines over time running yarn where the package.json uses all ^ type semver strings.

Also, sorry for hijacking this ticket. Happy to be directed to a better place for this discussion.

[wirmar]

I am running into the same issue, which results in my Build Tooling (create-react-app/react-scripts fork) breaking whenever I add a new dependency or run yarn install a second time. (When using yarn@0.23.4)

A workaround for me was downgrading yarn to v0.21.3.

[maximeg]

I'm running into this too.

Some context: I currently work from 2 different laptops (work & home, same config, same versions of yarn & node) on two branches (say "master" & "work" with work having some new deps not in master yet).
I switch a lot between the branches, running upgrades from "master" then merging "master" into "work", adding deps in isolated commits (with related handwork) in "master" then merging "master" into "work"...

Executing yarn (and even yarn upgrade) results in a dependency dance that is a bit annoying and breaks the "like bundle install" promise.

Example of change I see:

-lodash@4.12.0, lodash@4.x.x, lodash@^4.0.0, lodash@^4.0.1, lodash@^4.2.0, lodash@^4.2.1, lodash@^4.3.0:
+lodash@4.12.0, lodash@^4.0.0, lodash@^4.2.0:
   version "4.12.0"
   resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.12.0.tgz#2bd6dc46a040f59e686c972ed21d93dc59053258"
 
-"lodash@>=3.5 <5", lodash@^4.14.0, lodash@^4.15.0, lodash@^4.16.4, lodash@^4.17.2:
+lodash@4.x.x, "lodash@>=3.5 <5", lodash@^4.0.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.16.4, lodash@^4.17.2, lodash@^4.2.1, lodash@^4.3.0:
   version "4.17.4"
   resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.4.tgz#78203a4d1c328ae1d86dca6460e369b57f4055ae"

And on a later run (after switching back and forth from a branch, or switching of laptop (git pull && yarn), another change is made, a requirement changes from one version to another.

Here is my expectation on executing yarn: it should read from the yarn.lock (possibly from package.json to detect mismatch, but just to warn).

And given this part taken as example:

lodash@4.12.0, lodash@^4.0.0, lodash@^4.2.0:
  version "4.12.0"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.12.0.tgz#2bd6dc46a040f59e686c972ed21d93dc59053258"
  ...
 
lodash@4.x.x, "lodash@>=3.5 <5", lodash@^4.0.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.16.4, lodash@^4.17.2, lodash@^4.2.1, lodash@^4.3.0:
  version "4.17.4"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.4.tgz#78203a4d1c328ae1d86dca6460e369b57f4055ae"
  ...

• for package having dependency stated as lodash@4.12.0, lodash@^4.0.0 or lodash@^4.2.0, use version "4.12.0".
• for package having dependency stated as lodash@4.x.x, "lodash@>=3.5 <5", lodash@^4.0.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.16.4, lodash@^4.17.2, lodash@^4.2.1 or lodash@^4.3.0 use version "4.17.4".
  and have no other logic and do nothing more.

Now for generating this yarn.lock file, it seems it lacks a deterministic way of choosing what requirement is mapped to what version.

I expect when doing yarn add or yarn upgrade that a requirement chooses the highest possible matching version. Always the highest. That would be deterministic and prevent such a dance.

Now, looking at the "requirement line" of a version, it must have a deterministic order (alphabetical, logical somehow...)

From my example, after an upgrade resulting in adding a new version of lodash (say 4.17.5), the yarn.lock file should move to:

lodash@4.12.0:
  version "4.12.0"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.12.0.tgz#2bd6dc46a040f59e686c972ed21d93dc59053258"
  ...
 
lodash@4.x.x, lodash@^4.0.0, lodash@^4.0.1, lodash@^4.2.0, lodash@^4.2.1, lodash@^4.3.0, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.16.4, lodash@^4.17.2, "lodash@>=3.5 <5":
  version "4.17.5"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.5.tgz#ababababab"
  ...

and be stable (unless lodash is upgraded, or a dependency requirement on lodash changes).

What do you think ? Was that the plan ?

[bestander]

This got fixed in #3477 and released in 0.26 this Friday.
After that it should do a one-off dependency optimization and that is all

[calidion]

having the same problem with the latest version.

[bestander]

@calidion, could you post repro steps in a separate issue?