[rush] "rush update" fails if npm-shrinkwrap.json is present (sometimes)

[trusktr]

For anyone stumbling here, the workaround is to simply remove common/config/rush/npm-shrinkwrap.json, then run rush update, if you're using npm instead of pnpm or yarn.

Is this a feature or a bug?

• Feature
• Bug? I'm not sure, but I can other wise run npm install in each project just fine, just rush install is not working.

Please describe the actual behavior.

Both rush install and rush update fail with errors like

npm WARN tarball tarball data for @rush-temp/test@file:projects/test.tgz (sha512-Mc7KWHTDuwpcuBr3J4XbLHRINUOcRSjcCbzsJqBxEgY37Ma2ttP98RQxFG/svbU2F2xoU0rx/1NdT+NrWJgpiQ==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for @rush-temp/readem@file:projects/readem.tgz (sha512-kweTbInRMah92jf5yK8pKd/kv64n01c4Bu31BIaiLgYdQ8PaZIitHk2Qx7L7qCuGnr0MiRANkDrVwF7TjqbIPQ==) seems to be corrupted. Trying one more time.
npm WARN tarball tarball data for @rush-temp/infamous@file:projects/infamous.tgz (sha512-amm5Vq/+H6OvEc1OW2B2qc5cpmLZRzJd36QR2aVzj2gZtlwIBJdPmEJe5yKT25fhR+XA6FnYc9JsBWFr5g1qow==) seems to be corrupted. Trying one more time.
...
npm WARN tar ENOENT: no such file or directory, open '/home/trusktr/src/test-rush-project/common/temp/node_modules/.staging/@microsoft/rush-lib-c8592c0e/lib/cli/actions/AddAction.js.map'
npm WARN tar ENOENT: no such file or directory, open '/home/trusktr/src/test-rush-project/common/temp/node_modules/.staging/@microsoft/rush-lib-c8592c0e/lib/utilities/AlreadyReportedError.d.ts.map'
npm WARN tar ENOENT: no such file or directory, open '/home/trusktr/src/test-rush-project/common/temp/node_modules/.staging/@microsoft/rush-lib-c8592c0e/lib/utilities/AlreadyReportedError.js.map'
npm WARN tar ENOENT: no such file or directory, open '/home/trusktr/src/test-rush-project/common/temp/node_modules/.staging/@microsoft/rush-lib-c8592c0e/lib/utilities/AsyncRecycler.d.ts.map'
npm WARN tar ENOENT: no such file or directory, open '/home/trusktr/src/test-rush-project/common/temp/node_modules/.staging/@microsoft/rush-lib-c8592c0e/lib/utilities/AsyncRecycler.js.map'
npm ERR! code EINTEGRITY
npm ERR! Verification failed while extracting @rush-temp/readem@file:projects/readem.tgz:
npm ERR! Verification failed while extracting @rush-temp/readem@file:projects/readem.tgz:
npm ERR! Integrity check failed:
npm ERR!   Wanted: sha512-kweTbInRMah92jf5yK8pKd/kv64n01c4Bu31BIaiLgYdQ8PaZIitHk2Qx7L7qCuGnr0MiRANkDrVwF7TjqbIPQ==
npm ERR!    Found: sha512-Hs/jKW/C2q6/HJWRsflCfMAht745aIHORxzOEy754Toax/nCbwvI4impiTlAlmw5+Y9I2iDNqLQpWvrOMjJklQ==

What is the expected behavior?

No errors.

Reproduction

I've made two branches, in which the only difference is in common/config/rush/npm-shrinkwrap.json; one works, the other doesn't:

• doesn't work: https://github.com/trusktr/test-rush-project/tree/rush-issue-1706-not-working
• works: https://github.com/trusktr/test-rush-project/tree/rush-issue-1706-working

If this is a bug, please provide the tool version, Node.js version, and OS.

• Tool: rush
• Tool Version: 5.18.0
• Node Version: 13.6.0
• OS: macOS

[trusktr]

Hmmm, interesting: I started over from scratch, with rush init in a new folder, copied the three projects into a the new location's packages/ folder, using rush 5.18 this time, and added the three projects in rush.json just like before, and now it all works!

[apostolisms]

@trusktr Closing this since it doesn't seem to not be an issue anymore. Feel free to reopen if the issue shows up again.

[trusktr]

deleted

I updated the other project that isn't working to rush 5.18, but still errors, so that's not it.

I diffed the two folders, and these are the differences I noticed:

• common/temp/ folders are different, but I don't think there's any concern here as the errors would mean they end up in different state most likely. I started each without the temp folders before running commands.
• common/config/rush/npm-shrinkwrap.json contents differ in each project. I will try to remove it from the older project and start clean.
• the infamous project in the non-working repo has some dangling symlinks inside a build output folder, as a sideeffect of building and then removing files. It is nothing abnormal as these are re-written with ln -sf during build of the project. I wouldn't think that these cause any problems with rush, as they are not located in any folders that rush touches; they're in the docs/ folder.
• the node_modules folders inside the non-working repo projects are missing dependencies, but this is probably due to the error that is happening.

That's literally it! There are no other differences, yet one project fails with those errors!

EDIT: I cleaned up the repos, deleted all ignored files (temp folders, node_modules folders, etc), and the difference between both project is very minimal. In the following screenshot, the directory structure on the left if the repo that doesn't work, and the one on the right is the one that works:

As you can see, not much significant difference. You see dangling symlinks on the left, npm-shrinkwrap files differ, some prettier config files in the broken repo, and package-lock files at the root differ. Really nothing obvious here!

[trusktr]

@apostolisms I can not figure out why one repo is borked but the other is not. Let me try removing those differences from above to find the culprit, and then maybe this will be actionable.

[trusktr]

Alright! I reduced the diff to literally just npm-shrinkwrap.json,

After removing this file in the broken repo, it all started working!

So there's some sort of problem with how rush is handling npm-shrinkwrap. It'd be great to figure exactly what the difference is, so that other people won't run into this strange issue later.

I've made two branches, in which the only difference is in common/config/rush/npm-shrinkwrap.json; one works, the other doesn't:

• doesn't work: https://github.com/trusktr/test-rush-project/tree/rush-issue-1706-not-working
• works: https://github.com/trusktr/test-rush-project/tree/rush-issue-1706-working

(make sure to git clone --recursive, or use the git submodule commands to ensure that packages/infamous is cloned, or similar)

@apostolisms I hope that helps. Do you want to re-open this now that I have made the most minimal reproduction? I edited the OP to include the reproduction as well.

For anyone stumbling here, the workaround is to simply remove common/config/rush/npm-shrinkwrap.json, then run rush update.

[apostolisms]

@trusktr Thanks for the repro. I will reopen to investigate.

[trusktr]

I found a way to go from a good state to the bad state:

in a working repo, I ran rush purge, then on rush update it started having the problems again, and finally I was able to recover by first removing the npm-shrinkwrap.json file and running rush update again.

I'm not sure what the preconditions should be so that running rush purge will cause the repo to go to the bad state.

What we can try is:

• clone the good branch (linked above),
• rush update to initialize everything,
• run rush purge followed
• run rush update and hope now it has the error.

I'm not sure if this depends on npmjs.com packages having been updated since the last shrink wrap. Possibly if there are no updates to any packages, then there won't be any SHA changes.

[trusktr]

npm ERR! Integrity check failed:
npm ERR!   Wanted: sha512-kweTbInRMah92jf5yK8pKd/kv64n01c4Bu31BIaiLgYdQ8PaZIitHk2Qx7L7qCuGnr0MiRANkDrVwF7TjqbIPQ==
npm ERR!    Found: sha512-Hs/jKW/C2q6/HJWRsflCfMAht745aIHORxzOEy754Toax/nCbwvI4impiTlAlmw5+Y9I2iDNqLQpWvrOMjJklQ==

Why are the SHAs different? Where does it get the "Wanted" and "Found" values from?

[octogonz]

What version of NPM are you using? If I remember right newer versions aren't supported very well due to lack if interest from the community.

[octogonz]

See here for some background: #886

[trusktr]

Ah, I'm on NPM 6.13.6

[C-o-d-e-C-o-w-b-o-y]

I am getting this error with yarn when yarn.lock is present in common/config/rush/yarn.lock. Says my own package has an invalid signature, but it's my own package (local project, not in npm)!! @microsoftopensource pls fix, or let us do rush update --skip-integrity-check

[melanke]

Just want to let you know that this error still occurs, but thanks for the workaround, OP