poetry installing from private pypi (incl. CI/CD jobs)

[vlcinsky]

Table of Contents

1. Feature request: poetry installing from private pypi (incl. CI/CD jobs)
   1. TL;DR
   2. Background
   3. Requirements
   4. Proposed solution
      1. Named repositories ready to serve as sources
      2. sources as list of source names to use
      3. package without source name installs from any index
      4. How to express value of sources
      5. Preference of sources definition
   5. Open issues
      1. Terms repository, (poetry) source, index and pypi
      2. How to prevent use of public pypi
      3. poetry search ignores currently sources
   6. Conclusions

Feature request: poetry installing from private pypi (incl. CI/CD jobs)

For serious project development, usage of one or more private pypi indexes and flexibility to use different ones during CI/CD jobs is essential.

This proposal describes possible poetry modifications to allow that.

TL;DR

Reuse existing option to define named repositories in poetry global config.

Instead of expressing sources directly in `pyproject.toml` (what lacks flexibility when indexes has to be modified), allow specification of (repository) names to be used as sources.

Introduce env. variable POETRY_SOURCES which would be space delimited list of repository names to be used as sources.

Background

• Poetry related:
  • Poetry allows definition of named repositories in global config (config.toml). Optionally username and password for basic authorization can be specified (stored in auth.toml).
  • Poetry allows specification of named sources in pyproject.toml. This can reuse basic authorization parameters from auth.toml file.
  • There is draft PR of @sdispater to allow package specification to express name of pypi to install from.
  • Poetry allows upload of the package to an index, this is what repositories are used for.
  • There is issue #625 proposing to use global repositories as fallback sources (with PR, which was rejected).
• CI/CD related:
  • the same python package might be uploaded to or read from different pypi indexes during different phases of testing.
    • during development, volatile indexes can be used (which allow replacing already existing package version with another having exactly the same version spec)
    • during staging, non-volatile indexes can be used, which would reject attempt to replace already existing package version by another instance.
  • devpi-server is good example of private pypi suitable to development. It supports multiple indexes with options to be volatile, non-volatile, use bases pointing to other indexes etc.
  • real url of pypi index to install from may differ during different phases of testing, integration and deployment.

Requirements

• For poetry sub-commands such as install, add and search allow usage of alternative pypi url(s) without need to modify pyproject.toml file.
• Changing url of any pypi index shall be easy using environment variable(s).

Regarding the last item: It is fine, if this happens using one or more poetry calls via CLI passing value of env. variable on command line.

Proposed solution

Named repositories ready to serve as sources

Use existing configuration of repositories by poetry which allows to define name, url and optionally username and password.

Anyway, named repository definition does not make it automatically available as a source unless specified explicitly (see below).

sources as list of source names to use

pyproject.toml, global config.toml, command line parameter --source or env. variable POETRY_SOURCES would allow to define list of source names to use.

The names would refer to repositories named in global config. Order of names would be significant.

package without source name installs from any index

All packages, which do not specify explicit source name, do attempt to install from any source.

Note, that at this moment package does not allow stating source name, but this might come in future.

If a package states the index name, it can install only from source with given name.

How to express value of sources

1. Global poetry config

   Global poetry config file config.toml would introduce

   [settings]
   sources = ["private", "gitlab/prod", "javl/dev", "pypi"]
   

2. Command line argument

   Commands poetry add, poetry install and poetry search would allow multiple --source options specifying source name.

3. pyproject.toml source

   As of today, following would provide complete source definition in pyproject.toml:

   [[tool.poetry.source]]
   name = 'private'
   url = 'http://example.com/simple'
   

   In case url defined in global config would be intended for real use, missing url or providing empty string would mean "use the url defined globally":

   [[tool.poetry.source]]
   name = 'private'
   

   or alternatively

   [[tool.poetry.source]]
   name = 'private'
   url = ''  # use globally defined url
   

Preference of sources definition

As multiple places defining sources can be in place, following order of preferences and processing rules shall apply:

In short:

• if pyproject.toml source does not specify url and there is no globally defined repository with given name, fail processing.
• if source name is used and it is not found in global config, fail processing.
• sources from pyproject.toml are always used as most important
• first of following methods, CLI --source, env var POETRY_SOURCES or globally configured ~sources, may contribute additional sources.

1. Use all sources defined in pyproject.toml

   Whatever source appears in pyproject.toml shall be used. Use them in given order.

   In case any given source is using empty url, require existence of globally defined repository with given name and use that url. Fail, if such global repository is missing.

2. Use all sources defined on command line

   If --source name is specified on command line call, use it. This option ca be used multiple times.

   Use those names in given order and append them to names possibly read from pyproject.toml.

   If any ~source option was used, skip further processing of env. variable or globally defined sources.

3. Use sources from env. variable POETRY_SOURCES

   Note: if --source CLI option was used, env. variable is ignored.

   Value of POETRY_SOURCES is space delimited list of repository names to use as sources.

   If any of names specified are missing in global configuration, fail.

   Use all of these names and append them to names possibly read from pyproject.toml.

4. Use of globally configured sources

   Note: if --source CLI option or env variable POETRY_SOURCES were used, globally configured sources value is ignored.

   sources value read from global config is list of repository names to be used as sources.

   If any of given names are not defined as repository, fail.

   Append all source names to names possibly read from pyproject.toml.

Open issues

Terms repository, (poetry) source, index and pypi

There seem to be some redundancy. Could we merge some of the terms?

How to prevent use of public pypi

E.g. use --default-source and point to name of repository to use as default one. By default, it would use public pypi.

poetry search ignores currently sources

Conclusions

If there is some preliminary agreement on proposed solution, we may contribute PR.

We had another, simpler proposal, which assumed only one source url. Providing such url e.g. via devpi server is feasible but it would go against concept of named sources and repositories, which are already clearly visible concepts in poetry.

We are looking forward to make poetry essential part in (at least our) CI/CD jobs and scripts.

[robertlagrant]

It'd be good to be able to push env vars into the URL / auth parts of the file, so we can just set them in the surrounding context (local dev pull token; CI push token; whatever) and have it all Just Work™.

[vlcinsky]

Info: During our Friday testing we have realized, that content of poetry.lock shall be also properly handled as it contains urls of repositories to install from.

Another finding: using named repositories for uploading to indexes is necessary. However, named sources (as something to install from) could be probably simplified by using only single source. I am not sure, if such modification is best to discuss within this issue or to create a new one.

[drunkwcodes]

The named sources can't be simplified by that. It's fine by printing them out while requesting.

[robertlagrant]

However, named sources (as something to install from) could be probably simplified by using only single source.

@vlcinsky I don't understand this - how will this work if I have 20 PyPI dependencies and 3 Gemfury ones?

[vlcinsky]

@robertlagrant stay tuned - I will elaborate on this.

[vlcinsky]

@robertlagrant see related issue #1096

It explains possible reasons why it would be better not to use named sources for dependent packages and describes alternative approach.

I keep it out of this issue, which builds on existing approach.

[bjoernpollex]

@sdispater @brycedrennan @stephsamson It would be great to get some feedback on this feature overall. There's a ton of issues around this topic, and this one is the most comprehensive one in my opinion. Would it be worth going through related issues and pointing them here? Is the feature proposed here something that could find it's way into poetry? I'd be happy to work on this, but I feel because it's a pretty complex feature it would be nice to agree on a direction first.

[brycedrennan]

@vlcinsky @bjoernpollex I appreciate the thoughtful proposal here. As someone who uses CI and devpi its important to me that there are good workflows for this.

For the time being I'll be focused on bug fixes. I don't yet have the expertise to handle feature requests. @sdispater manages all the feature proposals. Its unknown when we'll be able to get to this. I know thats disappointing and I wish I had better answers. There are currently 29 open bug pull requests. Once we're caught up on open bugs I should at least be able to offer a review and thoughts on this.

[vlcinsky]

@brycedrennan it seems, your priorities are set very well. Thanks for all the maintenance effort, it is very important.

Looking forward to progress on this issue when the time comes.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Closing this issue automatically because it has not had any activity since it has been marked as stale. If you think it is still relevant and should be addressed, feel free to open a new one.

[vlcinsky]

Issue #1632 seems to be duplicate of this.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.