Avoiding custom repos when possible for significant faster locking

[omerfeinberg]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

The purpose of this suggestion is to significantly enhance performance for users who use both pypi and one or more custom repo for package discovery. 4X faster locks in my experience.

When poetry initializes a lock process it builds a pool of repos to query. it has two types of repositories: PypiRepository in case no other repo was set as default, and LegacyRepository for any user-defined repo.
During the dependency discovery process, poetry goes through the dependency tree, and for every dependency queries all repos for sub dependencies. as a result, if I set a custom repo from which I want to fetch a couple of specific packages, I will actually query it for available versions of every used package, resulting in hundreds of unnecessary calls.

My suggestion is to create a new property for repo called "Avoidable" (If anyone has a better naming suggestion I will appreciate it). When going over all available repos, every avoidable repo will be skipped in case a previous repo already returned an answer for the current package. It will be marked as false on default for backward compatibility purposes, although I believe in the future it should be true :)

Testing this approach in my day-job project (default pypi, custom gemfury, ±8 packages need the custom repo) gave incredible results. In version 1.18 adding this twick poetry's source code reduced locking time from 200-250 secs to 50-60. I've also tried applying in on current master (1.2.0a2), reducing times from 60 secs (which is already awesome btw. Great job whoever did this) to 15-18 secs.

The main drawback of this approach is when a custom repo contains custom versions of packages also available in pypi. in this case, the user can either specify from which repo this package suppose to come (This feature exists, right?) or just not use the avoidable flag.

I already have a working prototype of this which needs a bit of a cleanup, requiring PRs for both poetry and poetry-core. any opinions/suggestions before I go though with this?

[omerfeinberg]

A different approach would be a global configuration value which states whether the version solver should continue querying repos after the first valid response. defining only certain repos as "avoidable" may be over complicating things

[ofercaspi]

That sounds great, I'd love to have a feature of that sort, I've spent so much time waiting for locking :( . I would suggest the name "fallback" (instead of "avoidable"), and consider having it as an install/update flag too, and not only as a definition attached to the repo itself.

[neersighted]

A mergable redesign of sources is tracked in #6713. It covers this ask in a more robust/better designed way, more or less.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.