Git+SSH dependencies have subtle (yet critical) differences from git clone

[grahamc]

Reproduction steps:

1. Have a git repository which is private, and is a pip package.
2. Have a deploy key which can read the repository.
3. Confirm you can run git clone git@github.com:account/private-pip-package.git
4. Google around and discover that the instructions to use a git repo as a pip dependency boil down to very nearly "prefixing the repo with git+ssh:// and suffix it with #egg=private-pip-package."
5. End up with a pip dependency of git+ssh://git@github.com:account/private-pip-package.git#egg=private-pip
6. Try installing the package, but get a fatal: Could not read from remote repository. instead.
7. Run the gamut of tests (ssh git@github.com works, but ssh ssh://git@github.com fails ...)
8. Realize (after much hair-pulling) that instead of git+ssh://git@github.com:account/private-pip-package.git#egg=private-pip it should be git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip (if you didn't catch that, its github.com/account instead of github.com:account)

I would propose that a simple note in the failure message, or checking on input, or what have you, about the : vs. / would be quite helpful.

[Ivoz]

After checking back on the docs, I noticed you can use that form of uri by git+git@github.com:account/repo (the last example documented).

[grahamc]

Maybe that only works for public repos? This is for a private repository.

[mgasner]

While this works for pip install git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip, it doesn't help much when trying to use setup.py.

If you run setup(install_requires=['git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip'], ...), you'll get an error like error in package setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers.

If you instead run setup(dependency_links=['git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip'], install_reqs=['private-pip-package'], ...), you'll get an error like Could not find any downloads that satisfy the requirement private-pip-package (from package==1.2.3).

Perhaps you thought you'd be clever and do something like install_reqs=parse_requirements( 'requirements.txt')! Too bad, because #2244 (comment).

Is there really no lighter-weight, supported solution for requiring private repos than running your own PyPI?

[remohammadi]

If you instead run setup(dependency_links=['git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip'], install_reqs=['private-pip-package'], ...), you'll get an error like Could not find any downloads that satisfy the requirement private-pip-package (from package==1.2.3).

Pip wants to know the version before fetching (git clone) the dependency. If we specify the version after the project name in the egg string, in this hacky form: egg=<project name>-<version>, then there will be an item in _find_all_versions()'s returned list, and everything goes fine. So if we want to dependent our package on the latest commit in the branch STABLE-X of a VCS repository, we can do this:

install_requires=[
    'private-pip-packag',
],
dependency_links=[
    'git+ssh://git@github.com/account/private-pip-package.git@STABLE-X#egg=private-pip-9999999',
],

But, it's too hacky! I think these days it's so reasonable to have a project dependant on the latest version of a branch of some VCS repository, and there should be no need on specifying a dummy version number as a workaround for this purpose.

[ajmazurie]

Hi guys,
I am having the very same problem of not being able to install a specific commit of a Github-hosted project using setuptools' dependency_links option, and was wondering if this was still considered a bug or a misunderstanding of setuptools.

Say, for example, that I want to install a specific commit of the project paramiko-expect; for example, https://github.com/fgimian/paramiko-expect/tree/943630ac499284e6441854a9c4ae1e04301bfdd9

Here is what works and what doesn't, using pip 8.1.2 and setuptools 23.1.0:

• 🆗 pip install "git+ssh://git@github.com/fgimian/paramiko-expect.git@943630a#egg=paramiko-expect-0.2"
• 🚫 (in dependency_links in setup.py) git+ssh://git@github.com/fgimian/paramiko-expect.git@943630a#egg=paramiko-expect-0.2
• 🚫 (in dependency_links in setup.py) https://github.com/fgimian/paramiko-expect/archive/943630ac499284e6441854a9c4ae1e04301bfdd9.zip#egg=paramiko-expect-0.2
• 🚫 (in dependency_links in setup.py) https://github.com/fgimian/paramiko-expect/zipball/943630a#egg=paramiko-expect-0.2

Note that the setup.py file then has paramiko-expect==0.2 in its install_requires field, and that I tried all possible variants of paramiko-expect versus paramiko_expect. The setup.py file is then used by typing pip install ..

Is there something I'm missing here?
Best, Aurélien

[BertrandBordage]

@mgasner @ajmazurie Same issue here… I almost got crazy by spending 3 hours testing all the possibly imaginable combinations for this, and couldn’t find a single one working.

Pip is always trying to find the private module on PyPI, which of course doesn’t work. Instead, it should simply install all dependency_links before checking that the correct version of each install_requires package is installed, and otherwise try to download it on PyPI.

Of course, installing what I put in dependency_links using pip install works.
And obviously, dependency_links items are ignored when they don’t match an install_requires entry, so we can’t even ask for the setup script to always reinstall the same module without checking it was installed.

[xavfernandez]

If you want pip to take dependenc_links into account, you should use the --process-dependency-links.

[ChillarAnand]

@xavfernandez --process-dependency-links is deprecated?

[xavfernandez]

@ChillarAnand it is, but without any deprecation timeline and available replacement so this is still the way to go.

soo, there is no way to install a git+https://{url/to/repo}@{branch}#egg={package}-{version} url in setup.py? my understanding was that your module's dependencies should go into setup.py and your pinned-for-release dependencies should go in requirements.txt. But If I cant load first party dependencies in setup.py, this work flow cant be done.. what?

Edit, I suppose this workflow is also complicated by the fact that pip doesn't preserve the source uri of its package, so pip freeze doesn't actually produce an install-ready file, since the "dependency links" equivalent names have to be manually inserted into the file. :(

[wobeng]

Any update on this issue? having the same problem

[max-sixty]

@wobeng What problem are you having? The solutions above worked for me

[wobeng]

I have a private package A on github and it requires a private package B on github. When I try to install A, it tries to installs B but stop short with an error saying Could not find a version that satisfies the requirement package B No matching distribution found for package B

[max-sixty]

What's the exact situation? What's the string in setup.py you're using?

[wobeng]

@MaximilianR are you on slack?