Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] gzip archives (from git) have inconsistent checksums across Linux distributions #76

Closed
ThiefMaster opened this issue Apr 16, 2021 · 7 comments

Comments

@ThiefMaster
Copy link

ThiefMaster commented Apr 16, 2021

What / Why

This is very similar to #62 and possibly npm/cli#2846 but since it's happening reliably on the same machine (ie no architectural differences) in separate docker containers and comments are more likely to be overlooked, I preferred to open a new issue.

I can reliably reproduce this bug using ubuntu and archlinux Docker images using these two Dockerfiles (just for the sake of reproducing it easily) - simply run builds using docker build -f Dockerfile.XXX . and check the different lock files:

How

FROM ubuntu:focal
RUN set -ex && \
        apt update && \
        apt install -y curl git && \
        curl -fsSL https://deb.nodesource.com/setup_15.x | bash - && \
        apt install -y nodejs
RUN npm i -g npm@7.10.0
RUN mkdir /test
WORKDIR /test
RUN npm i --verbose 'github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af'
RUN cat package-lock.json
{
  "name": "test",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "dependencies": {
        "jqplot": "github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
      }
    },
    "node_modules/jqplot": {
      "version": "1.0.9",
      "resolved": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-X/WC4DGdoiLof0cK/nTywyNzBwTNsEwH7Ky6ndwn5SUgsNmZDNnfugNhpfMX1y3Jh+GG6O9UxSMaFH/3pcffHQ==",
      "license": "(MIT AND GPL-2.0)"
    }
  },
  "dependencies": {
    "jqplot": {
      "version": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-X/WC4DGdoiLof0cK/nTywyNzBwTNsEwH7Ky6ndwn5SUgsNmZDNnfugNhpfMX1y3Jh+GG6O9UxSMaFH/3pcffHQ==",
      "from": "jqplot@github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
    }
  }
}

FROM archlinux/base:latest
RUN set -ex && \
        pacman -Sy --noconfirm && \
        pacman -S --noconfirm nodejs npm git && \
        pacman -Syu --noconfirm
RUN npm i -g npm@7.10.0
RUN mkdir /test
WORKDIR /test
RUN npm i --verbose 'github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af'
RUN cat package-lock.json
{
  "name": "test",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "dependencies": {
        "jqplot": "github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
      }
    },
    "node_modules/jqplot": {
      "version": "1.0.9",
      "resolved": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-hMjKgDiIZ2RWZOe0wOUk9V1kWwyuvpNoqIoDT1hJ/1RmzKnYIfKM1BUPdJAo4gXr/LgmEF6GxGPZ1uXn7cfVBw==",
      "license": "(MIT AND GPL-2.0)"
    }
  },
  "dependencies": {
    "jqplot": {
      "version": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-hMjKgDiIZ2RWZOe0wOUk9V1kWwyuvpNoqIoDT1hJ/1RmzKnYIfKM1BUPdJAo4gXr/LgmEF6GxGPZ1uXn7cfVBw==",
      "from": "jqplot@github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
    }
  }
}

When running npm pack in node_modules/jqplot manually I also get the different integrity hashes, but when I gunzip those files the .tar files have the same checksum so it's clearly related to gzip producing different output.

On my Gentoo system I get the same hash as on archlinux; I just used arch in the dockerfile because it's faster than compiling nodejs manually on a Gentoo image ;)

ThiefMaster added a commit to ThiefMaster/indico that referenced this issue Apr 16, 2021
npm/pacote#76 caused issues for some developers when installing this
package from git
@luxaritas
Copy link

luxaritas commented Jul 13, 2021

I think we're running into this as well - we rely on github repo-based packages, and having installs fail due to integrity issues based on who's installing is rather problematic.

@ahuth
Copy link

ahuth commented Jul 22, 2021

I think we're running into this, too.

The package-lock.json is generated on linux dev environments. But pushes to Heroku fail for the integrity checksum mismatch.

@akselikap
Copy link

I just ran into this issue and exactly the same variant as @ahuth. It seems like there hasn't been any updates to this as I'm using the newest version of npm (7.22.0) and it's still occurring.

@nevcos
Copy link

nevcos commented Sep 13, 2021

Same happening between Windows and Linux machines, very frustrating:

npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"
# integrity:     sha512-qpMpLBvXY0w8J[...]J8drdjr3X9/2w==   # Windows 10 with NPM 7.21.1
# integrity:     sha512-4IWjabwqciNun[...]aXD5qGvf1Yrnw==   # Windows 10 with NPM 6.14.15
# integrity:     sha512-qpMpLBvXY0w8J[...]J8drdjr3X9/2w==   # Ubuntu WSL with NPM 7.21.1
# integrity:     sha512-pu/VZ6qKntixi[...]iPaYSOpTaEgrA==   # Ubuntu WSL with NPM 6.14.15

@jameskerr
Copy link

I have a package that contains a prepare script to build a go binary when installed. When I run npm pack <my git dependency> I get a different integrity value each time I run it on the same machine. With npm 6 it was stable.

Npm 6 -- stable

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm -v
6.14.15

Npm 7 -- different every time

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-/i5QW944neOZe[...]PlmRMHSvk0hZA==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-kh664omwauvXp[...]UDd6RbkQKYW9A==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-iZW1VW06yVaQG[...]eytlUvcwkkR+g==

$ npm -v
7.24.2

Note on reproducing: This is the public Zed repo so you can use the commands above, but you'll need go and make installed for it to succeed.

@wraithgar
Copy link
Member

npm/cli#2846 (comment)

@ahuth
Copy link

ahuth commented Jun 3, 2022

Can confirm this is fixed for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants