Support private pip server+index as a Config Variable

[greysteil]

From @skylerbunny on December 27, 2017 0:23

In the same way that 'Manage config variables' allows a user to add a private gem server or private docker registry, you'll probably want to allow the use of private PyPI libraries. For instance, one can be created with the https://github.com/devpi/devpi Devpi project.

If setting this up, the functional use of this variable would be to provide up to three additional options to the pip install command Dependabot uses to determine versions (See 'pip help install'):

  -i, --index-url <url>       Base URL of Python Package Index (default
                              https://pypi.python.org/simple). This should
                              point to a repository compliant with PEP 503
                              (the simple repository API) or a local directory
                              laid out in the same format.
  --extra-index-url <url>     Extra URLs of package indexes to use in addition
                              to --index-url. Should follow the same rules as
                              --index-url.
  --trusted-host <hostname>   Mark this host as trusted, even though it does
                              not have valid or any HTTPS.

For the purposes of Dependabot, I suspect it would be sufficient for almost any common scenario to allow the config variable input as so:

Language:
Python (PyPI)

Private PyPi server (PEP 503 'simple' repository URL):
[blank for URL entry]

[x] Use ONLY this Private PyPI server for version bumping
Disabling the above option allows Dependabot to check both your private and the global PyPI library at https://pypi.python.org/simple for version bumps; newest on either will take precedence

See pypa/pip#3454 : index URLs have no precedence. Newest version of a library by name wins, and same version on either means there is no particular library chosen, simply "one of them".

--

From a practical standpoint, the three configurations this would support amount to:
(Nothing in the config variable filled in:)
Current behavior. Dependabot checks https://pypi.python.org/simple .

Private URL filled in and radio box enabled:
Equivalent of pip --index-url [Private URL string] --trusted-host [private hostname substring] is used.*

Private URL filled in and radio box disabled:
Equivalent of pip --index-url [Private URL string] --extra-index-url https://pypi.python.org/simple --trusted-host [private hostname substring] is used.*

* If a private URL is supplied, --trusted-host must be assumed unless another radio box will be provided to explicitly use or not use it. I'd presume that if someone puts in a URL either with http or with an https certificate that does not have root validation, they know what they're doing. This could be warned about.

** Credentials supplied to a private PyPI server appear in the URL itself, as stated in pypa/pip#4315 . It's extremely unlikely IMHO that a Dependabot user is going to want to supply a private credential in the clear, in the URL. Even if they did, there's no reason it wouldn't work; it would just be a horribly insecure practice, but it's all that PyPI allows for authentication to an index. In any event, at least for now, it would require no work to be done that isn't taken care of by allowing a private PyPI index in the first place.

Copied from original issue: dependabot/feedback#69

[greysteil]

Awesome - been waiting for someone to ask for this, and the above is really clear. I'll get something implemented in the next couple of days.

Thanks masses for the feedback, and for describing what's needed so thoroughly.

[greysteil]

Just shipped a first version of this - take a look and let me know what you think! I want to put a bunch more validations in there to help people out, but it should be working as-is. 🎉

[greysteil]

From @skylerbunny on December 27, 2017 18:37

(Sorry, I'm realizing I don't have a lot of time to explain this Right Now because I'm discovering that finding the perfect answers isn't in easy reach for the moment):

You may wish to do trivial checks at submit time of the Config Variable as to the validity of an index, even if the index is empty (which it can be). That being said, what defines a valid index is available at https://www.python.org/dev/peps/pep-0503/ .

I think it would probably be sufficient to:

• If a trailing slash is not present on the URL, that it be added (a simple index must end with a slash to not involve a redirect)
• Once the above is checked (and added if needed), that the URL be validated for a return code of 200.

This presumes, of course, that your code doesn't have error checking at version bump time to assure that an index URL is valid, so this at least would eliminate the most obvious suspects.