[BUG] NPM v7 uses SSH instead of an explicit HTTPS for GitHub repos

[uhop]

Current Behavior:

When I use a git repository via an HTTP link NPM "takes liberties" with it, which breaks my build:

$ npm init -y
Wrote to /Users/eugene.lazutkin/Work/temp/package.json:

{
  "name": "temp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


$ npm i --save https://github.com/uhop/stream-chain.git

added 1 package, and audited 2 packages in 3s

found 0 vulnerabilities

It produces package-lock.json:

{
  "name": "temp",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "stream-chain": "github:uhop/stream-chain"
      }
    },
    "node_modules/stream-chain": {
      "version": "2.2.4",
      "resolved": "git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e",
      "license": "BSD-3-Clause"
    }
  },
  "dependencies": {
    "stream-chain": {
      "version": "git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e",
      "from": "stream-chain@github:uhop/stream-chain"
    }
  }
}

Note that https://github.com/uhop/stream-chain.git was replaced with github:uhop/stream-chain, which is probably OK in this case. But other two links (?) are rewritten from https://github.com/uhop/stream-chain.git to git+ssh://git@github.com/uhop/stream-chain.git, which is clearly bad.

The problem is that a build bot we use in similar situations can access private git repositories using HTTP, but not SSH for security reasons. It fails on an authentication. Rewriting https://github.com/uhop/stream-chain.git to git+ssh://git@github.com/uhop/stream-chain.git is not acceptable for that reasons.

The fix is relatively minor yet unpleasant: we have to replace npm ci with npm i, which takes more time and introduced instabilities with other dependencies.

Expected Behavior:

When running npm ci it should use the original URL with the HTTP authentication instead of SSH.

Steps To Reproduce:

See the description and do the same steps using git repositories (github only?) as dependencies.

Environment:

OS: Mac
Node: 15.7.0
NPM: 7.4.3

[ljharb]

Can you try with npm v7.5.2? I think this has already been fixed.

[uhop]

Just tried with exactly same results.

$ npm --version
7.5.2

package-lock.json is absolutely identical.

[uhop]

I think there is no error in the code.

Just to point it out explicitly:

• The dependency is set as https://github.com/uhop/stream-chain.git — note https:// part of it.
• In package-lock.json it is mentioned three times.
  • Two if them as git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e.
  • Note git+ssh://git@github.com part.
  • It definitely tries to access the repository using ssh rather than https protocol and fails.
    • Why is it important?
      • Those protocols have a different way to authenticate a user.
      • Our build environment is not set up for ssh, yet we can use https.

I don't know if the bug is specific to GitHub: adding https://github.com/uhop/stream-chain.git was transformed into github:uhop/stream-chain (please peruse my bug report above), and who knows what protocol it assumes. But not all repositories support ssh. We clearly have a bug on our hands.

I hope I explained the problem better now. Please do not hesitate to ask, if something is left unclear.

[uhop]

It came to my attention that this is a documented behavior that breaks private repositories and environments without ssh access (from https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major):

Git dependencies on known git hosts (GitHub, BitBucket, etc.) will always attempt to fetch package contents from the relevant tarball CDNs if possible, falling back to git+ssh for private packages. resolved value in package-lock.json will always reflect the git+ssh url value. Saved value in package.json dependencies will always reflect the canonical shorthand value.

Clearly this is not correct and should be fixed.

[plaa]

This is a breaking change in npm v7 and prevents anyone using even public packages without ssh keys.

We have made a fork of an npm package and host it as a public repo. We refer to it using a github: link. In v6 this github link was populated to package-lock.json as well, which worked fine. In v7 it is converted into git+ssh: which fails in our CI/CD pipeline as it doesn't have any ssh keys configured.

We don't want to add ssh keys to our CI pipeline just to fetch public packages, so I don't see any way we can upgrade. I'd be glad to hear of any workaround.

Specific example:

In package.json referring to https://github.com/Vincit/winston-azure-transport:

    "winston-azure-transport": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215"

Diff of package-lock.json between npm v6 and v7:

     "winston-azure-transport": {
-      "version": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",
-      "from": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",```
+      "version": "git+ssh://git@github.com/Vincit/winston-azure-transport.git#ed07d0685ca601638ab8ebcf660128848dd30215",
+      "from": "winston-azure-transport@github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",

Our CI now fails with:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@github.com/Vincit/winston-azure-transport.git
npm ERR! 
npm ERR! git@github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128

[isaacs]

This is tricky.

We currently keep the https url if auth is set in the url, because presumably you need that auth to access the repo.

The tricky bit is that using https without auth in the url will trigger a prompt to enter a username and password on the command line, which will fail (or at least be confusing and strange) in many cases where an install with ssh would work fine.

If the repo is public, and a git sha is specified in the spec that we're fetching, then we never hit it via git at all; npm will just fetch the tarball from the appropriate CDN url for the known host. However, if you do not have a git sha in the spec (typically, because there's no lockfile present), then we will have to do a git rev-list to figure out which sha we need to fetch, and that's where it runs into trouble.

Perhaps an approach that both works in build scenarios without ssh keys and for dev machines where ssh keys are present, while minimizing the scenarios where we need to prompt for a username/password for basic auth, would be to do the following for resolving any known-host git spec:

1. If the spec has a git sha, just fetch the tarball from the CDN, and we're done. No git commands needed. If that fails, continue (either a private package or an invalid git sha).
2. If the spec is git+https, and has auth specified, save the full git+https url, and only ever access via https.
3. Attempt to fetch the rev-list via the git+ssh url for the known host/repo. This will fail if ssh keys are not available, but at least will do so without confusingly prompting for user interaction, in the case of private repos where ssh keys are available. If that fails, continue.
4. Attempt to fetch the rev-list via the git+https url for the known host/repo. This will fail if auth is required for a private package. (Note: best to just close stdin immediately, so that it fails as quickly as possible.) But, it will succeed in cases where auth is not required and ssh keys are not available.

Maybe we could also add a config to tell npm which git interaction to prefer, thus swapping the order of (3) and (4) above? --git-prefer=(ssh-first|https-first|ssh-only|https-only), defaulting to ssh-first? Then you could set npm_config_git_prefer=https-only in the build environment, and have it never try ssh at all.

[Tallyb]

Having an all repo config is probably not the best idea, as packages come in all shapes and forms.
I think respecting the installation method provided during the npm install is probably the best approach. If anyone needs ssh or https, they should simply add the package this way.

[isaacs]

Having an all repo config is probably not the best idea, as packages come in all shapes and forms.

This is only relevant when pulling packages from known hosts like GitHub or GitLab, where the "shape and size" is fairly well understood. The difference is in the current environment capabilities. Ie, can it use ssh? should it fail if it can't get the repo via git+https, or should it fall back to ssh? should it use https first, or ssh first? These are all system-specific, not dependency- or repo-specific.

[uhop]

These are all system-specific, not dependency- or repo-specific.

Could these decisions be left for a developer, who provides explicit dependencies? Like if a URL starts with https:// it is, well, https, and if it is git+ssh:// then it should be, well, git over ssh? Just asking for a friend...

[uhop]

Having well-known hosts is admirable, as long as it is transparent, yet it smells like magic — an extra-bandwidth knowledge, which makes the system behave differently on unexpected factors. In my experience, it is a major source of support incidents and perceived bugs even when everything works fine.

[themightychris]

this magic is unacceptable, there are no two ways about it. Do what you want with a github: link but if I address a dependency explicitly with https:// or git+https:// then NPM needs to use the requested protocol

[vincentdesmares]

There is the exact same issue on yarn 1.X, and it's solved the same way, by adding the "git@" in the URL.

[hans2520]

There is the exact same issue on yarn 1.X, and it's solved the same way, by adding the "git@" in the URL.

The git@github.com solution doesn't work for private repositories, because it will actually attempt to authenticate with the "git" user.

We would actually need to put in our own dev account username in there to make this approach work, which is 100% not feasible.

Tested on the latest version of npm -- 9.8.1 -- and haven't upgraded past v6 because of this nonsense.

EDIT: because a fix for this doesn't look like this is coming any time soon and the last vresion where this works is out of support, have added this bash workaround and used where needed:

function configureGitForceHttps {
    git config url."https://".insteadOf ssh://
}

[baybal]

This is tricky.

We currently keep the https url if auth is set in the url, because presumably you need that auth to access the repo.

The tricky bit is that using https without auth in the url will trigger a prompt to enter a username and password on the command line, which will fail (or at least be confusing and strange) in many cases where an install with ssh would work fine.

If the repo is public, and a git sha is specified in the spec that we're fetching, then we never hit it via git at all; npm will just fetch the tarball from the appropriate CDN url for the known host. However, if you do not have a git sha in the spec (typically, because there's no lockfile present), then we will have to do a git rev-list to figure out which sha we need to fetch, and that's where it runs into trouble.

Perhaps an approach that both works in build scenarios without ssh keys and for dev machines where ssh keys are present, while minimizing the scenarios where we need to prompt for a username/password for basic auth, would be to do the following for resolving any known-host git spec:

1. If the spec has a git sha, just fetch the tarball from the CDN, and we're done. No git commands needed. If that fails, continue (either a private package or an invalid git sha).
2. If the spec is git+https, and has auth specified, save the full git+https url, and only ever access via https.
3. Attempt to fetch the rev-list via the git+ssh url for the known host/repo. This will fail if ssh keys are not available, but at least will do so without confusingly prompting for user interaction, in the case of private repos where ssh keys are available. If that fails, continue.
4. Attempt to fetch the rev-list via the git+https url for the known host/repo. This will fail if auth is required for a private package. (Note: best to just close stdin immediately, so that it fails as quickly as possible.) But, it will succeed in cases where auth is not required and ssh keys are not available.

Maybe we could also add a config to tell npm which git interaction to prefer, thus swapping the order of (3) and (4) above? --git-prefer=(ssh-first|https-first|ssh-only|https-only), defaulting to ssh-first? Then you could set npm_config_git_prefer=https-only in the build environment, and have it never try ssh at all.

I just encountered this behaviour too, to my big surprise, and wasted 1 day figuring it out.

At least on Github, private repos are also available through CDN, if you bearer token is set, but currently NPM doesn't provide an option to set HTTP auth header bearer token for the CDN. You can hardcode a token into dependency URL, but this is no go approach.

[roman-shandurenko]

Depends on the use case but this helped me in CI pipeline (github app token as GH_TOKEN env var is used for authentication but PAT should work too):

git config --global --remove-section url."ssh://git@github.com" &> /dev/null

Redirecting output to /dev/null helps to avoid error if section doesn't exist.

[diegogangl]

Is there any workaround for private repos? This use case is 1000% broken. Other than using an ancient version of npm? I can't believe this has been broken for +2 years at this point.

[dacevedo12]

I discovered this is specific to github due to auth being null in

cli/workspaces/arborist/lib/consistent-resolve.js

Line 30 in d04111d

return `git+${hosted.auth ? hosted.https(hostedOpt) : hosted.sshurl(hostedOpt)}`

. Tried with gitlab and worked OK

Perhaps this if more of a bug in https://github.com/npm/hosted-git-info

[surgiie]

This is happening to me on npm v9.

[diegogangl]

Here's the workaround I've been using for a while. I wrote a small script to replace the git+ssh to git+https in package-lock.json.

const fs = require("fs");
const path = require("path");

// I keep this file in a scripts folder
const packagelock = path.resolve(__dirname, "../package-lock.json");

fs.readFile(packagelock, "utf8", (err, data) => {
  if (err) {
    return console.log(err);
  }

  const result = data.replace(
    /\"resolved\"\:\s\"git\+ssh:\/\/git\@github\.com\/my\/repo\.git/g,
    '"resolved": "git+https://[USERNAME]:[TOKEN]@github.com/my/repo.git'
  );

  fs.writeFile(packagelock, result, "utf8", (err) => {
    if (err) {
      return console.log(err);
    } else {
      return console.log("Package lock was fixed");
    }
  });
});

I run this in the dockerfile before running npm install

RUN node scripts/fix_package_lock.js
RUN npm install --production

[kishorviswanathan]

This is only relevant when pulling packages from known hosts like GitHub or GitLab, where the "shape and size" is fairly well understood.

What about people running CI behind HTTP proxies ? Their only way to the internet is via the proxy and there is no way they would be able to use SSH. You cannot force SSH just because the Git host supports SSH, it might not work for everyone.

It has been more than 3 years since this issue was created and it is a shame that people are still forced to use workarounds.

[Azq2]

Dear NPM developers, are you okay?
Why are you breaking the ability to use NPM with public repositories?
The ssh+git mechanism is absolutely not suitable for this task, because requires a github account and configured authorization via SSH.

If I use "npm i https://github.com/...", obviously I want to use the https+git protocol. Not ssh.
But why isn't this obvious to NPM developers?

​Really sad situation.

[Azq2]

Yet another workaround

npm install https://github.com/USER/REPO/tarball/master

[iarna]

Dear NPM developers, are you okay?

Are there NPM developers any more? I presume so, it seems like there are still npm updates, but... I dunno, the folks assigned to this aren't actually at Github now afaik, and even then, I'm still somehow getting notifications about this, and I haven't worked on NPM since they decided to fire some of my staff for trying to unionize. =p

[piranna]

@iarna, that doesn't looks good, thanks for sharing.