Suggest alternatives for --process-dependency-links

[rodart]

Environment

• pip version: 19.0
• Python version: 3.5.2
• OS: Ubuntu 16.04

New pip release v19.0 break --process-dependency-links flag. That completely sucks because :

1. there was no information in which exact version this flag will be removed. There was a warning like 'Dependency Links processing has been deprecated and will be removed in a future release.' that we can see for a few years without pointing the exact date or version where it will be actually removed.

2. this feature is still using by a lot of engineers/companies. For example, it broke the whole build pipeline for all our python projects in our company. You just created a headache for thousands of engineers.

3. there is no clear explanation or tutorial about which alternative we should use instead of this flag

I would suggest to:

1. Give detailed explanation what should be used instead --process-dependency-links flag

2. Provide exact version where you are going to deprecate or remove some functionality. Especially in such a global project such as pip. Good examples are numpy and scikit-learn which mention in which versions they will deprecate some features.

[pradyunsg]

Please look at #4187 (comment).

(edit from a future @pradyunsg -- this is a better answer: #5898 (comment))

[pradyunsg]

The above issue was linked in the error message, however, I do think now that more explicit communication would have helped the situation. The deprecation messages should've mentioned the release in which the functionality will be removed.

I guess the ship has sailed here, but I'll go update pip's deprecation helpers so that something like this doesn't happen in the future.

[stinovlas]

It seems that #5898 has not been addressed yet, so there is no way to replace whole dependency links functionality right now, is it? Of course, there is still an option to keep pip < 19 installed until this is resolved.

[fake-name]

This exploded my CI pipeline yesterday/last night too. Maybe have a meaningful depreciation warning in the future?

Last I'd heard, dependency links had been un-depreciated (I don't track issues here when things work), but now passing --process-dependency-links doesn't just not work, but it causes pip to explode.

This sort of thing needs at least 3-6 months of a new warning on installs before it drops.

[pkucmus]

After this I ended up running my own DevPI and I store my private packages there, using it as my main index. Aside from fixing the problem and allowing me to remove --process-dependency-links from all my "build scripts" it gives me other advantages like not being completely reliant on PyPi.

[ghost]

I have a package which references many git repositories as dependency_links in its setup.py file. This package is used in many of my code pipeline (~40 different code libraries using this package to provide common functionality). Pipeline is now totally broken using pip19. Please can someone tell me what the correct way should be to get git repositories installed during a build using pip >=19 ? From what i read so far is this functionality just been removed??

[stinovlas]

@x00x70: If you don't need version specifiers (or you can depend on one specific version of the package), just delete dependency_links arg from setup function call and write the dependencies directly into install_requires (and others) according to PEP 508.

I.e. requests @ https://github.com/requests/requests.git@branch_or_tag

If you need version specifiers, you are out of luck with pip 19.

[jenstroeger]

@stinovlas, I was hopeful that your suggestion would solve my problem, but while the tag v1.3.2 exists in the private repo, foo@ https://token@github.com/jenstroeger/foo.git@v1.3.2 returns

  HTTP error 404 while getting https://token@github.com/jenstroeger/foo.git@v1.3.2
  Could not install requirement […] because of error 404 Client Error: Not Found for url: https://github.com/jenstroeger/foo.git@v1.3.2

The same error when I use @master or nothing at the end of the URL. Are private repos a problem here?

[stinovlas]

@jenstroeger: Private repo requires some form of authentication. pip basically runs git clone, so just copy the same url as when you are cloning your repo (or the output of git remote -v).

In your case foo @ git+ssh://token@github.com/jenstroeger/foo.git@v1.3.2 should work.

[yeraydiazdiaz]

I can confirm the above does work. However, I noticed that changing the git tag in the URL does not install the new version. pip will report the dependency as already met and keep the existing version of the package. Here's a minimal setup.py:

from distutils.core import setup

setup(
    name='pep508_test',
    version='0.0.1',
    install_requires=[
        'requests @ git+ssh://git@github.com/requests/requests.git@v2.20.0',
    ]
)

Running pip install -e . will install version 2.20.0 as expected. However changing the tag to v2.21.0 will not install the new version.

I've tried using requests==2.21.0 @ git... but pip returns the following error:

'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'@ git+ss'"

I'm not sure if it's a problem with pip or the grammar. PEP 508 does not show many examples on URL dependencies, the above being one and name [fred,bar] @ http://foo.com the other.

@pradyunsg any advice?

[stinovlas]

That's the lack of version specifiers I've been refering to few comments above. It was discussed in #5898, but I don't think that it was resolved yet. Hopefully, it will be addressed soon.

Edit: Also, there is #5780.

[pradyunsg]

@pradyunsg any advice?

Yea. We need to have a PEP update for allowing version specifiers. If someone wants to spearhead a resolution here, please start a thread on distutils-sig about such a change (#5898 (comment)) and once that is done, a PR would be awesome.

I can come to this a bit after we've resolved the PEP 517 situation so someone else picking this up would be great to get things moving quicker.

[jenstroeger]

Private repo requires some form of authentication. pip basically runs git clone, so just copy the same url as when you are cloning your repo (or the output of git remote -v).

In your case foo @ git+ssh://token@github.com/jenstroeger/foo.git@v1.3.2 should work.

@stinovlas, that doesn’t work. The token I use is my developer token, it’s not an SSH public key. So changing from https to your suggested ssh gives me

token@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

I could switch to a developer’s public key but I’d prefer to understand why the http approach worked with pip==18.1 and then broke with pip==19.0. Meanwhile, I’m rolling back pip.

[stinovlas]

Yea. We need to have a PEP update for allowing version specifiers. If someone wants to spearhead a resolution here, please start a thread on distutils-sig about such a change (#5898 (comment)) and once that is done, a PR would be awesome.

I've started a thread on distutils-sig. Thanks for your advice!

I'm not promising the PR just yet, but when the PEP issue is resolved, I'd like to chip in.

[stinovlas]

You can read my short summary of the distutils-sig thread at #5898.

[pradyunsg]

If there are any concerns wrt dependency links being removed in pip 19.0, please raise them in #5898.

I'm going to go ahead and close this issue.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.