Replace the secondary source type with more granular types

[neersighted]

Poetry currently will look up packages in every source, even when unnecessary due to a source = constraint on a dependency, or if the dependency was already found. This behavior often surprises users (though it was originally modeled on pip). Ostensibly, secondary is the solution to this -- however, it merely pushes the repository further back in the search order and does little to address the user complaints/confusion. As such, we should deprecate it and replace it with new options that better match user expectations.

I think, to preserve backwards compatibility, the long-term path forward is probably two different options (in my example, supplemental and private). I would propose something like the following, as an overview of behaviors:

• No options means that the repository overlays PyPI, but we will still fall back to PyPI if necessary (e.g. suitable for internal mirrors), pip --extra-index-url-style. This is known internally as primary, and is the current behavior.
• default = true means that the repository replaces PyPI, pip --index-url-style. This is the current behavior.
• supplemental = true means that this source is only consulted after a lookup in the default (implicitly PyPI unless configured otherwise) and primary sources fail.
• private = true means that the repo will only be considered for packages that are explicitly configured with source =, and should be preferred by users for private packages to avoid dependency confusion attacks.

secondary = true would be kept around as a deprecated option (likely with a warning), and would maintain the legacy behavior of being searched exhaustively for backwards compatibility.

Originally #5984 (comment)

[neersighted]

#6669 represents incremental progress towards this goal, and the author is interested in implementing this proposal.

[MasterNayru]

Would it be fair say that a contributing factor to issues that users have when managing multiple repositories is that Poetry assumes the worst case scenario when it comes to how to obtain metadata from non-PyPI repositories? Lookups to PyPI are done through APIs and happen way quicker than the HTML parsing that happens with the other repositories. Obviously, Poetry has gone down that route to ensure that things work and don't just explode because some backend didn't whatever APIs PyPI is currently exposing.

Just seems to me that while querying repos unnecessarily is worth avoiding, a real pain point is how these other package repositories end up being utilised.

[neersighted]

I did not reference it here as it is out of scope for this issue; however, yes, that is a problem. However, it is an ecosystem problem:

For the second part, Poetry must download (and often even build) packages to gather metadata and recursively resolve dependencies. In order to reduce this work (and achieve the same performance we have with PyPI, which has a non-standard API that we can gather some of this information from), the accepted PEP 691 and PEP 658 standards need to be implemented by each of your custom repositories. The good news is that they are very deliberately fully backwards compatible with the existing API, and will not require any action from end users.

The bad news is this will take some time -- PyPI has yet to implement 691, though a PR has been long in the works. Once PyPI gains support, we will likely implement this in Poetry and unify all of our source handling code to no longer special-case PyPI.

(from #4113 (comment))

In short, PyPI implements a non-standard API that has had to make breaking changes with no warning in the past, and that they intend to remove. It has served Poetry's use case so far and validated the need for this in the ecosystem, but it is not implementable by other package sources for a variety of reasons (ranging from the data model being coupled to warehouse internals and incorrect in the general case, to detection being problematic).

There is a standards-based solution and path forward; Poetry will have to wait patiently with everyone else for gradual adoption.

[b-kamphorst]

#6669 represents incremental progress towards this goal, and the author is interested in implementing this proposal.

Indeed I am. The refactor is "hairy" (not my words), but it appears to be a nice way to get acquainted with the Poetry code and contribute.

I'm now at the point where I write tests that express the desired behavior and found that the private case needs to be made more explicit. Specifically: if a package is configured with source and targets a private repository, what does that mean for finding that package's dependencies?

Currently, such dependencies are not bound to the source, meaning that they may be retrieved from any configured repo including the source repo. Perhaps the most natural option is to, when we introduce private repos and a package's source is set to that private repo, its dependencies can be retrieved from all repositories including the private repo that is the package source (and no other private repos). However, there are alternative scenarios as well. For example, only might suggest that dependencies may not consider any private repo.

The first suggestion seems most appropriate. My private packages have private dependencies, so I'd expect that to work just fine. Still, I'd like to hear your thoughts before I make assumptions.

[neersighted]

Regarding the correct model for transient dependencies when requested from a source = with private = true, I am personally inclined to take "the easy way out" and require the user to list them as top-level deps with private-transient-dep = "*" as the suggested mechanism -- that way constraints are driven by their sibling dependencies.

I find this less surprising as all dependencies in Python are "peer dependencies" in other ecosystems, and thus we don't have to answer the question of where a dep comes from when it's a mutual dependency of multiple top level deps with disparate source = markers.

However, if we must propagate source =, I agree that recursively allowing lookup in a private = true source (of course additive, to solve the mutual tree issue) might be less surprising in the common case. It's just that the design is harder to reason about, harder to document, may interact in surprising ways with non-Poetry tools/expectations, and may complicate things in the complex cases.

Poetry has generally taken a stance of "if you care about any details of your transient dependencies, well, now it's a top-level dependency," which is maybe constructed backwards vs. the reality of the Python ecosystem (most existing tooling and standards are built around flat lists of deps and not trees). In my mind, it makes more sense to users who are new to Python packaging and better expresses Poetry's goals of bringing useful ideas from other ecosystems to Python, while remaining compatible/a good citizen.

[dimbleby]

Can I suggest setting out what you think the key use cases for configuring repositories are, and spelling out how those use cases would be met by this proposal?

With any luck that'll validate that this is a great solution. With even more luck, it'll help someone to realise that their case is not well served, or that the model can be simplified without hurting anything.

The cases that come to mind for me are

• just use pypi, don't make me think
• only use a mirror, I care about supply chain security and cannot rely on public servers
• use an additional repository (or repositories) for specific packages only, eg I want to get pytorch from their repository but everything else from pypi

in my small world I don't know the use case for "internal mirror with fallback to pypi", but perhaps it's a thing people want. (Maybe those with slow or limited internet connections have reasons to prefer local mirrors but are willing to fall back to pypi?)

[kkozmic]

The cases that come to mind for me are

One more that is crucial to my team:

• use pypi for most, and a private repository only for our internal packages. Internal packages always as a rule define source = "our-internal-repo". We would expect internal packages to only be looked up in our internal repo, and all other packages to only be looked up in pypi

[strangemonad]

@kkozmic that is also our primary use case. Specifically, poetry is starting to get unbearably slow in our larger projects. Our private repositories are hosted in gitlab and when gitlab doesn't find a package it returns with a redirect to pypi. This means any local caching is defeated because the requested and returned urls differ

[b-kamphorst]

I've made a draft PR to facilitate the discussion. High-level feedback is very welcome (may save future revisions)!

[b-kamphorst]

We regularly use an internal mirror with fallback to PyPI. Our packages are developed internally and made public with some delay (and not all versions are released externally). During development of package A, I need to be able to use the latest features of package B -- possibly not available on PyPI yet. When the new versions packages A and B are released externally, users of package A should just use the version of package B that is available on PyPI.

[b-kamphorst]

secondary = true would be kept around as a deprecated option (likely with a warning), and would maintain the legacy behavior of being searched exhaustively for backwards compatibility.

@neersighted is supplemental behaviour actually breaking for secondary? If secondary sources are considered after all other (primary, default) sources, then the resolution behaviour should be identical (apart from making / timing of queries). If not, what am I missing?

[jgentil]

This is an excellent proposal. In particular, the private option will mitigate so many headaches we have currently, where updating and fresh installs are a total pain because of having to use Azure Devops' PyPI implementation that is terribly slow.

[vmgustavo]

Is there any palliative solution?

[vmgustavo]

Related to this issue and #7204:

If I try to force poetry to use the private source as the primary source and try to ignore the default pypi it still compares the results of pypi.org with the private source.

<private-pypi-name>: Response URL https://pypi.org/simple/ipykernel/ differs from request URL <private-pypi-url>/ipykernel/

[radoering]

Since the implementation of default sources does not match the documentation (see #7430 for details) and changing the implementation so that sources configured as default do not have the highest priority can be considered a breaking change, I wondered if we need the option to specify a default source at all?

Actually, I think we do only need an option to disable PyPI. This need not to be tied to setting a flag on a specific source. #7430 is a draft (with some TODOs) to deprecate default and add an option to explicitly disable PyPI. Are there any opinions whether it's better to disable PyPI by configuring another source as default or to disable PyPI via an explicit command, whichs sets a property (e.g. disable-pypi) not tied to another source in pyproject.toml?

[radoering]

In the last maintainer meeting we revisited the proposal, refined it and came up with a slightly differnt design alternative. I'll try to summarize it here, so we can decide if the alternative or the (revised) original proposal makes more sense.

General decisions (same in both proposals)

Secondary sources (secondary = true) will be deprecated. The behavior will be kept as is until removal: Secondary sources are queried after the default PyPI source if there are other (non-secondary, i.e. "primary") sources defined. If there are only secondary sources they are queried before the default PyPI source. Secondary sources are always queried if the source is not explicitly specified for a dependency (even if a suitable package/version has already been found in other sources).

Setting a default source (default = true) will be deprecated since it's not required as a priority but only to disable the default PyPI source. For that purpose an explicit option will be introduced (see #7430 for details). The behavior of the deprecated default will be kept as is until removal: Defining a default disables the default PyPI source and the source marked as default becomes the source with the highest priority.

Without secondary = true and default = true only one of the existing source types remains. These unlabeled sources (internally called "primary") are queried before the default PyPI source and each primary source is always queried (even if a suitable package/version has already been found in another source).

With only one remaining source type we need new options to define the following behaviors:

• A source that is only queried if no suitable package/version has been found in other sources.
• A source that is only queried if it is defined explicitly for the dependency.

In the following I'll summarize the revised original proposal and the new variant omitting the deprecated features for more clarity.

Original proposal

Introduce a new property for sources called priority (or type) that can have the values primary (same as if not set, behavior as described before), supplemental or explicit.

Supplemental sources will only be queried if no suitable package/version has been found in other (primary and supplemental) sources before. The order of supplemental sources is the order of occurrence as in pyproject.toml.

Explicit sources are only queried if they are defined explicitly for a dependency.

New proposal

Introduce two new flags: serial and explicit. Explicit is the same as before. The priority/order of sources is only defined by the order in the pyproject.toml. Unflagged (aka "primary") sources are (kind of) queried in parallel (not really but each source is queried even if a package has been found in another one). If a source is labelled as serial, it is only queried if no suitable package/version has been found before.

In contrast to the original proposal, sources are not sorted by groups first, but divided into groups by the order of occurrence in the pyproject.toml. Let's consider the following example:

• A
• B
• C (serial)
• D
• E (serial)

That would result in A and B being queried first "in parallel". If no suitable package/version is found in A or B, C and D are queried "in parallel". If no suitable package/version is found in C or D, E is queried. (If a package/version is found in one group the following groups are not queried.)

Closing thoughts

In both proposals, the default PyPI source (if not disabled) has the lowest priority and is only queried if no suitable package/version has been found in any other (non-explicit) source. This should help to avoid dependency confusion attacks.

What do you think? Which proposal makes more sense? Which one is easier to teach? (The names of the new options are not carved in stone yet and may be replaced if there are more suitable names.) Is there any use case not covered by one (or even both) proposals?

[b-kamphorst]

Thank you for the detailed overview and proposal. Perhaps the one thing that can be clarified is what the resolution order implies (perhaps also to add to the docs in the PR that handles the final proposal)?

To the best of my understanding (please agree / correct). If several sources are queried for a certain dependency in the same 'batch' (that includes the current "primary", default and secondary which are all queried before their respective packages are inspected), then all package versions for the dependency from all queried sources are returned by poetry.repositories.repository_pool.RepositoryPool.find_packages. There are only two callers of this function, namely poetry.puzzle.provider.Provider.search_for and poetry.version.version_selector.VersionSelector.find_best_candidate. In both cases, the packages are sorted in order of decreasing package version. As a result poetry always tries to satisfy all constraints with the package that has the highest version and this package is retrieved from the first source that yielded it. So if a secondary source yields version 1.0.1 and all other (primary, default) sources yield version 1.0.0, then the secondary source takes the win. If another source that provides the same version is queried earlier in the batch, then that source takes the win. A next batch (new behaviour) is considered only if none of the sources in the previous batch has a package that meets the current constraints.

As for the general decisions part: you have my full support in both decisions.

Regarding the other part, my thoughts are the following. The new proposal is strictly more generic than the original proposal -- if every supplemental source is flagged as serial than we obtain the first approach. However, it also seems to be harder to teach and more error-prone on the user side. The behavior needs manual adjustment of pyproject.toml as the implicit behavior cannot be provided through CLI. If I as a user expected a package to be retrieved from source D but it is retrieved from source A, the implicit behavior makes it harder for me to understand how the flag in source C is significant in the process. Suitable debug logging can help but we can not actively help the user to get this right. As such, I lean towards the first proposal UNLESS there is a clear use case that requires the second proposal.

Looking forward to others' thoughts.

[ralbertazzi]

Hi agree with @b-kamphorst: the first solution seems easier to understand and - while less generic - it also feels less "over-engineered". Did you ever receive issues that the first proposal would not solve but the second one would?

[ralbertazzi]

For anyone having the same issue of @kkozmic , I developed a simple opinionated plugin that makes sure that:

• Dependencies with the source specifier are only fetched from that source (Poetry already ensures that)
• 👉 Dependencies without the source specifier that have a result from the default source (pypi) are not searched in other repositories

You can find it here, feedback is welcome!
https://github.com/ralbertazzi/poetry-private-repo-plugin

[radoering]

While writing down the two approaches, I already had the feeling that the new proposal was too elaborate. Thanks for the confirmation. I think we will stick (more or less) to the original proposal.

#6713 (comment) shows that there is something we did not yet consider: People might have different requirements considering the priority of the default PyPI source. Some users want other sources only be looked up if there is nothing on PyPI, other users want PyPI only be looked up if there is nothing in other sources. Thus, it might make sense to make the priority of PyPI configurable.

[a-recknagel]

Just adding one more voice to the pool of "having a private flag on a source would be great".

Regarding the correct model for transient dependencies when requested from a source = with private = true, I am personally inclined to take "the easy way out" and require the user to list them as top-level deps with private-transient-dep = "*" as the suggested mechanism -- that way constraints are driven by their sibling dependencies.

Works for me, because if things get complicated I'd probably just set up a "pypi mirror plus private packages" and use that for everything, which solves the issue as a whole. Controlling the lookup and fallback order through clever setting of ever more granular flags for both sources and dependencies seems difficult and error prone.

[radoering]

Short status update:

The first step is done: #7658 has been merged into master. Thus, the next minor version of Poetry will introduce explicit package sources, which are only searched if they are explicitly configured for a dependency with source = ....

IMO, the next step is #7801, which will allow to configure the priority of PyPI. In its issue description, I also described my target image. (#7801 will supersede #7430.) Opinions are welcome.

[jaklan]

@radoering I have a question about that part:

In a future version of Poetry, PyPI will be disabled automatically if there is at least one custom source configured with another priority than explicit.

I don't really understand why this exception is only relevant to the explicit priority, but not to the supplemental one. If I add some source as a supplemental, it means I expect something different to be the primary one, which is also stated in docs quite clearly:

Package sources configured as supplemental are only searched if no other (higher-priority) source yields a compatible package distribution.

If I don't specify any primary source directly, imho it's quite obvious to assume it should be still PyPI.

Also, the scenario in which a) we don't have any primary sources b) we have the supplemental ones c) PyPI is set as explicit / supplemental should be then treated as a wrong configuration and raise an error (e.g. You have to configure at least one primary source).

---

Btw, the current docs are quite confusing:

• there is no description what primary sources are and how they are different from the supplemental ones - especially that information:

  each primary source is always queried (even if a suitable package/version has already been found in another source)

  is really missing, together with a note what happens if a higher version of package is found in a lower-priority primary source:

  • if higher package version wins, it means that note:

    Within each priority class, package sources are considered in order of appearance in pyproject.toml

    is not really true

  • if order of sources wins, that raises a question how to achieve the above behaviour (== look for the highest version in a set of sources)

• similar issue with supplemental sources: if we define multiple supplemental sources and package is not found in any primary one - are they all queried as well or maybe they are queried in order of appearance in pyproject.toml? What takes precedence: sources order or higher package version?

• imho we should already mark the default source as deprecated - docs in the current state really encourage to use default:

  • primary sources are not documented, so default seems as a "promoted" approach

  • it's always mentioned as a first solution to disable PyPI

  Both these points don't make too much sense if we want to get rid of default soon. Instead of that, we should promote disabling PyPI by configuring it as explicit and defining at least one primary source, as that's going to be the target approach anyway (with an annotation that the first part wouldn't be needed anymore when automatic PyPI disabling is enabled in a future version).

[radoering]

I don't really understand why this exception is only relevant to the explicit priority, but not to the supplemental one.

It has a technical background. We need at least one non-explicit source. Otherwise, we'll get a "no versions for package xyz found" error which would be quite obscure.

If I don't specify any primary source directly, imho it's quite obvious to assume it should be still PyPI.

Your reasoning makes sense. We probably should change it.

Also, the scenario in which a) we don't have any primary sources b) we have the supplemental ones c) PyPI is set as explicit / supplemental should be then treated as a wrong configuration and raise an error (e.g. You have to configure at least one primary source).

Technically, that's not an issue because if nothing is found in higher priority sources (which would be always true in this case), supplemental sources are searched. However, from a teaching perspective, I agree with you. If we want to change it we should print a warning first to avoid a breaking change (and change it to an error later).

• there is no description what primary sources are and how they are different from the supplemental ones

PRs with docs improvements are always welcome. 😄

• if higher package version wins, it means that note:
  Within each priority class, package sources are considered in order of appearance in pyproject.toml
  is not really true

That's still true. If the same version of a package is found in two sources the higher priority wins.

• similar issue with supplemental sources: if we define multiple supplemental sources and package is not found in any primary one - are they all queried as well or maybe they are queried in order of appearance in pyproject.toml? What takes precedence: sources order or higher package version?

In that case, at first, only the first supplemental source will be searched. Only if no versions fulfilling the constraint are found in the first one, the second supplemental source will be searched, and so on.

• imho we should already mark the default source as deprecated

We can't (don't want to) deprecate default before we we make this warning become true: "In a future version of Poetry, PyPI will be disabled automatically if there is at least one custom source configured with another priority than explicit."

• Instead of that, we should promote disabling PyPI by configuring it as explicit and defining at least one primary source, as that's going to be the target approach anyway (with an annotation that the first part wouldn't be needed anymore when automatic PyPI disabling is enabled in a future version).

That is an alternative we also considered. However, we decided that it's better to keep default and deprecate it as mentioned above so that users don't have to add an explicit PyPI (that can be removed later) if they don't want to use PyPI at all. With the chosen approach, users only have to change their configuration once (when we deprecate default). With the alternative, users have to change their configuration twice. (Strictly speaking, they don't have to remove the explicitly configured PyPI but IMO it would be weird to keep it.) This would feel like a back and forth to me.

[jaklan]

@radoering thank you for the quick answer!

Technically, that's not an issue because if nothing is found in higher priority sources (which would be always true in this case), supplemental sources are searched. However, from a teaching perspective, I agree with you. If we want to change it we should print a warning first to avoid a breaking change (and change it to an error later).

Fully agree with that approach. As you said - technically it is correct to have supplemental source(s) without any primary sources, but that could be really confusing for end-user from a conceptual perspective to trigger PyPI disabling as a side effect then. I believe the assumption of at least one (either explicit or implicit PyPI) primary source would improve general user experience.

That's still true. If the same version of a package is found in two sources the higher priority wins.

In that case, at first, only the first supplemental source will be searched. Only if no versions fulfilling the constraint are found in the first one, the second supplemental source will be searched, and so on.

To sum up then:

• primary sources are queried all together and the highest package version wins; if the same package version is found in multiple sources, the higher-priority source wins
• supplemental sources are queried one by one, so the first found package version always wins

Is that right? If yes, I wonder if that's actually the expected behaviour for users, as that's a bit "inconsistent". I could imagine some people wish to query primary sources one by one, and vice versa - to query the supplemental ones all together. Maybe we should think about making that behaviour configurable for all types of sources (so also the explicit ones, as that would be useful if we want to support multiple per-package source constraints in the future)? For example:

poetry config sources.[primary | supplemental | explicit].priority [version | order]

# version - we query all sources of given type and select the highest package version
# order - we query sources one by one until we find any version of a package

We can't (don't want to) deprecate default before we we make this warning become true: "In a future version of Poetry, PyPI will be disabled automatically if there is at least one custom source configured with another priority than explicit."

With the chosen approach, users only have to change their configuration once (when we deprecate default). With the alternative, users have to change their configuration twice. (Strictly speaking, they don't have to remove the explicitly configured PyPI but IMO it would be weird to keep it.) This would feel like a back and forth to me.

Yeah, I get your point, I also had the same thoughts, just assumed redundant, but not-breaking, explicit PyPI would be less affecting than deprecated default - but I agree it doesn't really matter so much as default was already used in the past versions, so user action is needed anyway. Imho docs about primary sources and a quick note that default is not yet deprecated, but it will be very soon, would do a job here and significantly clarify both current and future state of sources.

PRs with docs improvements are always welcome. 😄

I don't want to overpromise, but when I have the full picture (so when we discuss the above points), maybe I will find some time in the foreseeable future to prepare at least initial PR.

[radoering]

Is that right?

In principle, yes. Strictly speaking, it's not always the highest version but the highest version that fulfills all constraints.

I could imagine some people wish to query primary sources one by one,

In that case, they can configure one source as primary and all others as supplemental since that's the same.

and vice versa - to query the supplemental ones all together.

We considered making it more elaborate in #6713 (comment) but refrained from doing so since there were no real-world use cases.

Maybe we should think about making that behaviour configurable for all types of sources (so also the explicit ones, as that would be useful if we want to support multiple per-package source constraints in the future)?

Supporting multiple per-package source constraints is a new feature request. If you really have a use case for this, please create a new issue (if there is no such issue yet), so that we can discuss the consequences there.

Imho docs about primary sources and a quick note that default is not yet deprecated, but it will be very soon, would do a job here and significantly clarify both current and future state of sources.

Makes sense.

[jaklan]

@radoering

In principle, yes. Strictly speaking, it's not always the highest version but the highest version that fulfills all constraints.

Sure thing.

In that case, they can configure one source as primary and all others as supplemental since that's the same.

Right, good point.

We considered making it more elaborate in #6713 (comment) but refrained from doing so since there were no real-world use cases.

Yep, I agree querying primary sources one by one is more important use-case, but as mentioned above - it's actually achievable, so then we can simply observe if there's a real need for querying the supplemental ones all together and don't take any actions as for now.

Supporting multiple per-package source constraints is a new feature request. If you really have a use case for this, please create a new issue (if there is no such issue yet), so that we can discuss the consequences there.

That was actually just an example why we could make sources "priority" configurable for all types, not only primary and supplemental 😉 But if we talk about that functionality per se - personally I don't have such a need, however - I could imagine a scenario when it's useful (e.g. you want to add PyPI or another source as the "backup" one, but only for specific package(s), not all).

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.