The --frozen-lockfile command-line option does not work

[ddgenome]

The --frozen-lockfile command-line option does not work as advertised. It behaves the same way as when the option is not supplied, modifying the yarn.lock.

Here's an example:

$ yarn --version
0.23.4
$ git status -s
$ yarn --frozen-lockfile
yarn install v0.23.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
success Saved lockfile.
✨  Done in 6.99s.
$ git status -s
 M yarn.lock

Do you want to request a feature or report a bug?

bug

What is the current behavior?

The yarn.lock can be overwritten when --frozen-lockfile is provided on the command line.

If the current behavior is a bug, please provide the steps to reproduce.

Have a package.json that includes transitive dependencies with different versions. Sometimes, yarn will opportunistically collapse multiple dependencies on the same package into a single entry in the lock file if they can be satisfied by a single version. I'm not sure what triggers this. #579 contains a more thorough discussion.

Here is the link to a Travis CI build that exhibits the behavior: https://travis-ci.org/atomist/travis-rugs/builds/228523519
Here is the call to yarn that ended up with the yarn.lock being changed: https://github.com/atomist/travis-rugs/blob/c8b2dddfd129ae836a30864ec410ca9f01ff535e/.atomist/build/travis-build.bash#L68

What is the expected behavior?

yarn will make due with what is in the yarn.lock if it satisfies everything in package.json or fail if it can't.

Please mention your node.js, yarn and operating system version.

$ node --version
v7.9.0
$ yarn --version
0.23.4
$ uname -a
Darwin mbp.local 15.6.0 Darwin Kernel Version 15.6.0: Fri Feb 17 10:21:18 PST 2017; root:xnu-3248.60.11.4.1~1/RELEASE_X86_64 x86_64

[henryptung]

Same here. Yarn is also annoyingly shifting dependency versions slightly during the build, which breaks the stable-dependency check we have, even when --frozen-lockfile is used:

-object-assign@4.1.0:
+object-assign@4.1.0, object-assign@^4.0.1:
   version "4.1.0"
   resolved "<private NPM repository>/object-assign/-/object-assign-4.1.0.tgz#7a3b3d0e98063d43f4c03f2e8ae6cd51a86883a0"

-object-assign@^4.0.1, object-assign@^4.1.0, object-assign@^4.1.1:
+object-assign@^4.1.0, object-assign@^4.1.1:
   version "4.1.1"
   resolved "<private NPM repository>/object-assign/-/object-assign-4.1.1.tgz#2109adc7965887cfc05cbbd442cac8bfbb360863"

Versions (CircleCI Ubuntu 12.04 node):

$ node --version
v6.9.5
$ yarn --version
0.23.4
$ uname -a
Linux box106 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[ddgenome]

I have reported the shifting dependencies here: #3315

[mysterycommand]

@ddgenome @henryptung shouldn't the command be yarn --pure-lockfile?
https://yarnpkg.com/en/docs/cli/install#toc-yarn-install-pure-lockfile

[ddgenome]

@mysterycommand --pure-lockfile is a different option with a different meaning.

From the yarn --help output:

    --pure-lockfile                   don't generate a lockfile
    --frozen-lockfile                 don't generate a lockfile and fail if an update is needed

To my understanding, the intent of --pure-lockfile is to have yarn operate as it normally would, but just not write out the yarn.lock file. The intent of --frozen-lockfile is to install things exactly as expressed in the yarn.lock, failing if those modules do not satisfy package.json.

[mysterycommand]

@ddgenome, thank you for that. --frozen-lockfile isn't listed in the documentation for the install command, so I thought maybe it was an old/renamed flag.

[bestander]

This is where it should fail but does not https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/install.js#L340.

Help debugging and a PR are welcome

[gpoole]

The integrityError I'm seeing is LOCKFILE_DONT_MATCH, but missingPatterns is empty. In my case it seems to be because Yarn isn't changing the lockfile due to missing entries, it's just rearranging stuff. I've put up a failing test and hopefully a working fix in #3497.

[rarkins]

Am I missing something, or does this bug mean that yarn's main value proposition (deterministic, repeatable installs) is now broken as a result? Is there any way to ensure a team still gets identical installs?

[taoeffect]

Hey, there's NPM 5 out. :)

[BYK]

Working on this. Located the reason and place to fix. PR on the way after I figure out the tests part.

[BYK]

Oh wait I missed @gpoole's patch. Still, working on this :)

[rarkins]

@BYK could you link to the commit or release its fixed in?

[BYK]

@rarkins - 7515772

[patrickmarabeas]

Is there a reason why --frozen-lockfile fails, rather than warning on version conflicts?

[workmanw]

I'm left confused by reading through this issue. I've found that version mismatches of yarn between our CI server and individual dev machines to cause yarn.lock files to be left with subtle little changes. What I believe was referred to as "range collapsing".

All I want to do is yarn install --use-exactly-what-is-in-my-lockfile. I originally thought that was --pure-lockfile because the name implies that. Then I learned it means the opposite and I should use --frozen-lockfile. But that throws:

yarn install v0.27.5
[1/4] Resolving packages...
error Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.

How can I direct yarn to install exactly what's in the lockfile and not modify it?

[BYK]

I understand the confusion between --pure-lockfile and --frozen-lockfile. From what you've explained here, I think you need --frozen-lockfile. That causing yarn to fail means your CI and development versions of yarn are not fully compatible in terms of their resolution algorithms and you should update one or downgrade the other to match them for a consistency guarantee.

[workmanw]

@BYK Thank you for your response. I understand where you're coming from, but in reality it's going to be painful to coordinate the specific version of yarn on 6 developer machines, Travis-CI, our own CI server, what's required for OSS projects we contribute to, etc.

It would really be great if I could just direct the yarn cli to use exactly what's in the lockfile, don't update it, don't make any new decisions, don't throw exceptions due to mismatches, just use what was already decided upon.

[BYK]

It would really be great if I could just direct the yarn cli to use exactly what's in the lockfile, don't update it, don't make any new decisions, don't throw exceptions due to mismatches, just use what was already decided upon.

The lock file only determines the resolved versions. The resolved tree differes between yarn versions and it is codified in the algorithm itself. See kittens' post:

Even though Yarn hoisting may differ between versions we still make very strong guarantees around hoisting when the same version of Yarn is used.

npm 5 has stronger guarantees across versions and has a stronger deterministic lockfile, but Yarn only has those guarantees when you’re on the same version in favor of a lighter lockfile that is better for review.

That said I think I have a solution for you: https://yarnpkg.com/blog/2016/11/24/offline-mirror/#did-you-know-that-yarn-is-also-distributed-as-a-single-bundle-js-file-in-releaseshttpsgithubcomyarnpkgyarnreleases-that-can-be-used-on-ci-systems-without-internet-access

I've had a similar problem when I was working back at Disqus and we ended up checking in one of these bundles into the codebase (it's realtively small) and aliasing yarn to that checked-in version forcing everyone (CI and devs) to use the same yarn version.

Let me know if this helps you resolve the issue.

[workmanw]

Hey that's kind of awesome. I'll give that some consideration, thank you!

One addition that might really make this awesome would be if you could list yarn in the devDependencies section of your package.json file and the yarn-cli would just invoke the version installed in node_modules if present, rather than the global one. This is how ember-cli works and it's really a god send. With yarn it would be slightly trickier, the cli would have to first install the yarn package, then defer to that package to install the rest of the deps.

Either way, thanks again!

[BYK]

One addition that might really make this awesome would be if you could list yarn in the devDependencies section of your package.json file and the yarn-cli would just invoke the version installed in node_modules if present, rather than the global one.

Get out of my mind! I was actually thinking about something similar 😀

This is how ember-cli works and it's really a god send. With yarn it would be slightly trickier, the cli would have to first install the yarn package, then defer to that package to install the rest of the deps.

This is interesting and I'm okay trying to mimic an existing, well-established behavior. That said may be instead we can define yarn in scrips and point it to your local bundle file so when yarn sees that, it uses that one automatically? This avoids the download step and makes things a bit simpler.

I'd really encourage you to go over to https://github.com/yarnpkg/rfcs and try submitting a short RFC for this to see what others think. I'd love to ship this with 1.0 of we can get it ready.

Either way, thanks again!

My pleasure!

[rarkins]

@BYK so is it correct to conclude the following, or have I misunderstood?

Although it is essentially mandatory to coordinate/sync the version of yarn between teams to get deterministic behaviour, there is not currently any standard or convention for how to define which version yarn a repository uses?

I tried pinning the version of yarn in package.json > engines and using a shell script to awk it to feed to a yarn install script for developers, but was informed by a user that the result was that any end user not running the exact same version of yarn could then not install the package, so I reverted it.

[BYK]

@rarkins

Although it is essentially mandatory to coordinate/sync the version of yarn between teams to get deterministic behaviour, there is not currently any standard or convention for how to define which version yarn a repository uses?

Not really. I've just mentioned this in my earlier comments. The convention is to check-in the build yarn.js file into the repo unless you are using something like Chef to sync everything across all devices.

I liked your approach but don't understand the concern around

but was informed by a user that the result was that any end user not running the exact same version of yarn could then not install the package, so I reverted it.

Wasn't that the point then: to lock yarn versions and enforce everyone to use the exact same one?

[rarkins]

Wasn't that the point then: to lock yarn versions and enforce everyone to use the exact same one?

Yes, but it also locks versions for anyone attempting to install it globally as a CLI tool. Perhaps that's a bug/unintended side effect?

What I want is to make sure that anyone/anything (collaborators, CI, etc) that interacts with the yarn.lock is doing so with a coordinated yarn version. What I don't want is to restrict end users who just want to install it. i.e. there needs to be a way to pin yarn for development but not for use/install.

The package in question - quite relevant to yarn itself btw - is renovate. Here's an example of attempting to install one of the older versions before I removed yarn from the engines field:

❯ yarn global add renovate@8.30.4
yarn global v0.27.5
[1/4] Resolving packages...
[2/4] Fetching packages...
error renovate@8.30.4: The engine "yarn" is incompatible with this module. Expected version "0.24.5".
error Found incompatible module
info Visit https://yarnpkg.com/en/docs/cli/global for documentation about this command.

I don't think users should need to know/run --ignore-engines, if that works.

The only doubt I have is because I publish this module with npm not yarn. By any chance does yarn strip itself out of the engines field when it publishes? If so then that's something I could live with, but if not then I can't see how it's feasible to pin yarn version in package.json > engines without too many side effects.

[BYK]

@rarkins - I see your point. I'd still suggest checking in the yarn bundle into the repo and instruct people to use that instead of their global yarn. If you think that's not a great idea may be you can submit an RFC detailing your ideal solution (may be make yarn install the specified version automatically, locally and defer itself to that?): https://github.com/yarnpkg/rfcs

I think this may be quite interesting. //cc @yarnpkg/core

[rarkins]

@BYK after a few more minutes for the idea to bounce around, I think it's as simple as "yarn should not include yarn version in published package.json" or maybe "yarn should ignore yarn engines field when installing". Given that yarn.lock is not meant to be published, I can't see what benefit is derived from a yarn engines field passing through the publish stage. This can presumably be done quickly and with less planning than any "ideal" solution that auto-installs correct yarn versions as has been discussed above.

Do you foresee any problems with that idea?

[arcanis]

A few:

• The principle of least surprise would be broken. A field would behave differently from the others with no clear reason. People would be more likely to think this is a bug than an intentional behavior.

• Other package managers would have to adopt the same convention (and since I doubt they would, this alone makes this solution not applicable).

• We're toying around with exposing a set of API that packages can use. In this situation, enforcing Yarn's version is important.

If you want everyone to use the same Yarn version, you need to put the same Yarn version to everyone. Why don't you just put the single-file Yarn release inside your repository? That's essentially what we do at Facebook to make sure that we're all using the exact same Yarn version at any single point in time, commit-wise.

Now, what could be interesting would be to have another package on top of the regular yarn, let's say multiyarn, that would use the repository Yarn if available (for example by reading a field in a .yarnrc file), or fallback to another Yarn binary otherwise. Currently not on our roadmap, but if you're willing to experiment I'd be happy to see the result.

[rarkins]

Your solution of checking in the package manager may still be right, but I don't agree with most of the logic below. Reasoning follows:

The principle of least surprise would be broken. A field would behave differently from the others with no clear reason.

Yarn broke that principle in the first place though by enforcing engines by default, while "other package managers" did not. Therefore it seems flawed to now use this principle as reasoning to fix this problem.

Also consider this: the way it currently works, I'd say that using a pinned version of yarn in the engines field for any package that is published to npm is impossible in practice. Which means that pinning yarn version may as well be an unsupported feature, if it can't really be used. So essentially you are worrying about breaking an unusable feature. Search github.com for any repository using a pinned yarn version in their package.json engines and which isn't private - I got tired after paginating through about 20 pages of results and not finding any.

Other package managers would have to adopt the same convention

But we are talking about yarn's own engine field, not other ones. I wouldn't expect other package managers to care/enforce yarn's engine version the same way I don't expect yarn to enforce npm version or any new "foonpm" package manager version.

We're toying around with exposing a set of API that packages can use. In this situation, enforcing Yarn's version is important.

But aren't there are multiple ways this could be done, including an enforceEngines option like npm used to have?

Why don't you just put the single-file Yarn release inside your repository?

I might be missing something, but that seems like such a backwards approach for a package manager. Every other package in the world should be semantically defined but for the package manager itself.. commit a compiled file from another repo? It doesn't sit well with me even if that's what I'll probably end up doing.

[rarkins]

By the way, I don't have a problem with yarn enforcing the node version, which was the main topic of discussion in #1102. My point of linking to that was just to show that doing things differently - literally including the engines field - hasn't stopped yarn before.

[BYK]

Yarn smashed that principle to pieces in the first place by enforcing engines by default, while "other package managers" did not. It seems a bit flawed to now be claiming compatibility of behaviour with other managers is somehow sacred while before it was not. In fact we're only having this issue because yarn did not adopt the principle for this very topic. Discussion: #1102

I think the surprise part is not changing the whole behavior but your suggestion is to make it only ignore this for yarn versions.

Also consider this: the way it currently works, I'd say that using a pinned version of yarn in the engines field for any package that is published to npm is utterly impractical. Which means that pinning yarn version may as well be an unsupported feature, if it can't really be used. So essentially you are worrying about breaking an unusable feature. Search github.com for any repository using a pinned yarn version in their package.json engines and which isn't private - I got tired after paginating through about 20 pages of results and not finding any.

This feature may become useful for others too since yarn itself is essentially a dependency for the project. I really suggest trying to craft an RFC around this since this may help many other people if we can reach to a consensus about a solution.

But we are talking about yarn's own engine field, not other ones.

Not really. engines field is in package.json which is supposedly an "open standard", as in any potential package manager can use it as described.

I might be missing something, but that seems like such a backwards approach for a package manager. Every other package in the world should be semantically defined but for the package manager itself.. commit a compiled file from another repo? It doesn't sit well with me even if that's what I'll probably end up doing.

It felt weird to me too at first but the problem with syncing package manager versions is, you can't install anything without a package manager, include the package manager itself. So it has to come from or controlled from somewhere else. The repo itself looks like a pretty good place since it would be ready along with all other files you need for the project that you can't download from a package registry. Again, the alternative is to use Chef or a similar config management tool.

Also, I sense some anger/frustration in your tone. Did we do something to upset you? If so, how can we make it up to you and avoid it in the future? :)

[rarkins]

I think the surprise part is not changing the whole behavior but your suggestion is to make it only ignore this for yarn versions.

Right. I don't think it's unreasonable at all for yarn to treat the yarn engines field however it wants to, especially considering that there is no universal package manager approach to enforcing engines as a whole. At least, there isn't since yarn adopted strict enforcement by default.

I really suggest trying to craft an RFC around this since this may help many other people if we can reach to a consensus about a solution.

I just wanted to make sure we got past the "we have a problem" stage and to a "we have a possible solution". Or is RFC fine for an open "issue-like" problem discussion?

Not really. engines field is in package.json which is supposedly an "open standard", as in any potential package manager can use it as described.

Open or not, the engines field is obviously vague if yarn can adopt such different approach to npm. Further, package.json may be an open standard but "the way a package manager manipulates the package.json before publishing it to the registry" is not. i.e. if Yarn were to strip its own engines field out of the package.json the same way it essentially strips the yarn.lock out of the package too.

Let's say I run a service called "release automator" and the preferred way to configure it is by adding a "release-automator": {} section to package.json. Now if my service were to strip out its own configuration from the published package.json before publishing then it would hardly be violating any "open standard", right? I view yarn and engines>yarn manipulation as the same.

Also, I sense some anger/frustration in your tone. Did we do something to upset you? If so, how can we make it up to you and avoid it in the future? :)

Sorry, not my intention to convey any upset/anger and I have edited the first paragraph of my previous comment to reduce that impression.

[BYK]

At least, there isn't since yarn adopted strict enforcement by default.

Fair point.

I just wanted to make sure we got past the "we have a problem" stage and to a "we have a possible solution". Or is RFC fine for an open "issue-like" problem discussion?

I think we have identified the problem: there's no documented or automatic way to ensure yarn version compatibility across users of a project. I think this may go away with the release of 1.0 since we can then ensure that all major yarn versions to be compatible with their resolution algorithms and then you can put a rule like "yarn": "^1.1.0" in your engines section. We can also add something to our docs or publish a dedicated blog post talking about this potentially common issue and advise people to put the yarn.js binary into their projects. Or we create an RFC for another solution, such as yarn automatically downloading the proper version of itself for the project and the deferring to that along with the ability to defer to a previously checked-in and declared local version. This way you can have a global yarn with a different version than your project.

Open or not, the engines field is obviously vague if yarn can adopt such different approach to npm. Further, package.json may be an open standard but "the way a package manager manipulates the package.json before publishing it to the registry" is not. i.e. if Yarn were to strip its own engines field out of the package.json the same way it essentially strips the yarn.lock out of the package too.

I really find this approach dangerous and quite opaque. Although yarn differs about how it interprets the engines field, it is both well-documented and immediately visible (as a failure) where as if yarn silently stripped the yarn declaration in the engines field when publishing, people may not notice it for a very long time and have different assumptions.

Let's say I run a service called "release automator" and the preferred way to configure it is by adding a "release-automator": {} section to package.json. Now if my service were to strip out its own configuration from the published package.json before publishing then it would hardly be violating any "open standard", right? I view yarn and engines>yarn manipulation as the same.

If your release-automator field was well-established and commonly used, I'd say it would be safe to just start stripping it suddenly.

Sorry, not my intention to convey any upset/anger and I have edited the first paragraph of my previous comment to reduce that impression.

No worries. I'm just making sure we don't break any hearts here :) Also thanks for the edit!

[jamesdube]

Was having the same error with Travis, I upgraded my node version from 9 to 14 and it now works. Might be useful for someone else

[gabegorelick]

I upgraded my node version from 9 to 14 and it now works

This probably fixed the issue since upgrading the version of Node brought along a newer version of Yarn. Make sure you're not using a version of Yarn from before this was fixed.