[BUG] package-lock.json integrity value for git dependencies depends on the architecture (Apple Silicon M1 differs) #2846
Comments
feross
added
Bug
|
I've included both the Intel and M1 gzipped tarballs here, if that's helpful: Archive.zip |
|
I have both M1 and Intel mac, let me know if I can be of help. Here's the intel response: Here's the M1 |
|
A friend linked me this, and I just wanted to point out that relying on compressed data to match on all machines given the same uncompressed input is rife with problems. Implementations of Zlib exist that use hardware acceleration instructions, and they do not take care to ensure that each accelerated version will come up with the same compression as every other implementation. As an example, there is a Zlib implementation that uses x86 SSE4.2 instructions to accelerate pattern matching, and ARM doesn't have anything like that, so this is an x86-specific code path. It does not find the same exact dictionary matches, so won't come up with the same results as the pure C implementation. |
I also felt like this design decision might have this issue, even with the |
darcyclarke
added
Priority 1
|
@feross thanks for filing this - we'll take a look right away (apologized it wasn't triaged quicker) |
|
This bug had absolutely stumped me since december 2020, regarding this dependency tree: I created the Heroku support and I gave up trying to figure out the problem and for that GraphQL API I ended up downgrading to npm v6. Running with Node.js v15.12.0 and npm v7.6.3: In macOS v11.2.1 (Intel chip): In Ubuntu 20.04.2 LTS (Heroku, not sure what chip): At first I noticed the file permissions were created differently for the dependency installed locally ( This is a critical, show-stopping bug; how long until it's fixed in a stable npm release? Is there a workaround in the meantime, that does not involve downgrading npm v7 to npm v6? One gross possibility is not committing a lockfile and let Heroku generate one each deployment; the checksums can be whatever that way, since they're not compared against anything. Obviously that would have undesirable stability and security implications. |
|
i absolutely agree the root issue here is generating and comparing integrity hashes against a gzipped result. this patch to pacote forces the compression level to its highest setting, which on the OS, platform, and node versions i tested against (listed in the linked PR) did give me consistent results. it's definitely not "this is fixed forever" quality, but i'm hopeful it's enough to unblock folks while we determine a better approach. |
|
@nlf or @isaacs please reopen this issue; it has not been fixed: |
|
Had this issue when doing "npm ci" in a github workflow environnement, "npm update" right before "npm ci" did the trick for me PS: dependabot was creating different a shasum for a npm package (github private repository) |
|
Can we get this reopened please? It's clearly not fixed: npm/pacote#76 |
|
Getting this issue with The problematic entry in "tether": {
"version": "git+ssh://git@github.com/atlassian/tether.git#bf85430889b5231fbe5b383416cce6281225bf06",
"integrity": "sha512-/LwlT+VkWwl2WLu8PcceylDVJ+RlqnTVM6s3vMphdNysTvJOpWgJjA7HKM2ZaXiyWYgJvECUGiMuEhmJlkw8FA==",
"from": "tether@github:atlassian/tether#amd-with-global"
},The Previously I was running the x86_64 build of Today is the first time pushing to Heroku since switching to the native arm64 build of |
|
We've talked about this internally, and I think the consensus is that we really should not be tracking integrity for any dependency that gets built on the client, especially git dependencies that already benefit from git's comprehensive content hashing. (cc: @nlf) |
Sounds reasonable! |
|
Before an actual fix is available, here's a hacky oneliner that attempts to remove the checksums for git dependencies from A brave soul could run a proper version of this as a pre-commit hook. |
|
as of npm@8.5.2 we no longer compare the hash of git dependencies against an integrity string found in a package-lock.json. newly installed/updated git dependencies should not store an integrity field, though existing integrity strings may remain behind. if one of these is found, npm will log a warning telling you that the integrity value for the git repo was ignored. if you like, you can hand edit your package-lock.json and remove the offending line. |
It appears that different machines can produce different hashes for git-based dependencies, so the npm team moved to completely remove integrity checksums for them. Apparently these checksums were based on gzipped archives, which are not guaranteed to be binary identical for the same inputs across different CPU architectures. There is still some cryptographic integrity defense as the dependency is pinned to a git commit and that relies on the entire previous history of the repo, as discussed in the later parts of this issue on npm. npm/cli#2846
It appears that different machines can produce different hashes for git-based dependencies, so the npm team moved to completely remove integrity checksums for them. Apparently these checksums were based on gzipped archives, which are not guaranteed to be binary identical for the same inputs across different CPU architectures. There is still some cryptographic integrity defense as the dependency is pinned to a git commit and that relies on the entire previous history of the repo, as discussed in the later parts of this issue on npm. npm/cli#2846
…tion SHA integrity hashes for the downstream depedency Toast UI is built without React but we are depending on a wrapper published by the devlopers of the library. Both the underlying project and the react wrapper are customized. Removing the integrity SHAs because of this issue in NPM. npm/cli#2846
@nlf Running npm 8.12.1, running |
|
@luxaritas I could be mistaken, but I think that although the NPM CLI v8.5.2 no longer verifies the integrity of packages installed from |
|
The change of behavior mentioned here states that:
It seems though that npm behaves differently when installing packages locally from a tarball produced by The local integrity checksums are made from the compressed tarballs which might be architecture-specific, as was mentioned here. |
Prior to this version, the @webgpu/types dependency-by-revision won't work because the integrity hash is system-dependent. Starting with 8.5.2, integrity checks are skipped for git dependencies (though integrity hashes are still generated, despite the comment in the thread). See: npm/cli#2846 (comment)
Prior to this version, the @webgpu/types dependency-by-revision won't work because the integrity hash is system-dependent. Starting with 8.5.2, integrity checks are skipped for git dependencies (though integrity hashes are still generated, despite the comment in the thread). See: npm/cli#2846 (comment)
|
This is indeed a deeper problem with |
Prior to this version, the @webgpu/types dependency-by-revision won't work because the integrity hash is system-dependent. Starting with 8.5.2, integrity checks are skipped for git dependencies (though integrity hashes are still generated, despite the comment in the thread). See: npm/cli#2846 (comment)
Current Behavior:
The
package-lock.jsonintegrity value seems to depend on the OS/architecture. Take the following git dependency which specifies a commit hash:The integrity value produced is different on these OSes/architectures:
sha512-DnBTbDDxd9/9mwPehyraeuRTbNEqbWLcAdE3GC1trdBWWwKnkWsaU/X6mVLIKKB/IYWmG+cnL3ihg/Ql/rW5kg==sha512-T3ZWOM1TT+Ch/splApkEe1HwktWs+n/iHvDvtIGEI+4xuMGHite6mMujuNd8sen49ofLP/PxzprQMSPJK8APww==Expected Behavior:
The integrity value should not be different on Apple Silicon (M1 chip) machines.
Steps To Reproduce:
Run
npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"and inspect the integrity value from an M1 Mac. Node.js was installed from Homebrew usingbrew install nodeand theamd64version was installed.Also... @jhiesey and I dug into this a bit and found that the tarballs fetched from the GitHub CDN are exactly the same on M1 and other architectures, byte-for-byte. Same for the ungzipped tarballs – they are the same byte-for-byte. What differs, though, is the gzipped tarballs (
.tar.gz) files. Those appear to have substantial differences when viewed in a hex editor.Environment:
The text was updated successfully, but these errors were encountered: