Reduce resolver rounds by an order of magnitude

[pradyunsg]

This ensures that we're not trying "too hard" to resolve a dependency graph. Combined with the recent improvements to resolvelib, this should ensure that the resolution process takes a limited amount of wall clock time for complex graphs.

Toward #11480

[pfmoore]

In general, I'm +1 on this, but with my RM hat on, is this appropriate for 23.1, or do we need to do anything to avoid the risk of breaking complex installs with the lower limit? While I know that no-one tests with github main, I sort of feel that we'd have been less exposed if we could say it's been in main for a month or so - but a last minute reduction feels more rushed.

I'm not going to block this PR, but we should consider how we'd handle possible issues (I would be fairly unhappy if we needed to do a fast bugfix release just to revert this). If people are comfortable that the risk is low, then that's good enough for me. Maybe @notatallshaw could test some of the complex installs he's been checking with the reduced limit? I for one would appreciate that.

[notatallshaw]

Maybe @notatallshaw could test some of the complex installs he's been checking with the reduced limit? I for one would appreciate that.

Sure, I can check later this week, and I'll see if I can come up with an even more problematic requirements file.

I'm neutral on this change, because there isn't an understandable relationship between a given set of requirements and what the max rounds should be set to.

To me the only 2 numbers that have a clear understanding of what they mean are 0 and infinity, if infinity is set then Pip should find a solution if one exists (assuming no other limits) and if 0 then no backtracking should be done and the user has to fix their requirements/constraints.

The main pro of this change seem to be most related to CI tools to exit early and save energy / cost, as a user can exit the process at any time. Though a secondary pro would be for new users to Pip/Python not waiting several hours for an error.

The cons are what if a specific set of requirements needed just a small number of more rounds to solve and the requirements are complex enough the user is expecting them to take time to resolve, if resolution is allowed to complete then user can use the resolved requirements to improve their requirements going forward, but the max rounds error produced isn't particularly useful (however hopefully #10937 helps a little bit when the max rounds error happens).

[pfmoore]

One thought here - maybe pip install --dry-run --report should have a larger number of rounds, as presumably the user is willing to wait longer to get output that they can then use to create a pinned requirements file?

[uranusjr]

IIRC there were requests making this configurable. Although I don’t generally like this, maybe it is a good idea to introduce an undocumented environment variable for this while we test the water? This can help us figure out with the user whether increasing the number is actually a good solution for various situations.

[pradyunsg]

IIRC there were requests making this configurable.

The request to make this configurable is in #11480 -- which basically boils down to "things take too long". This PR reduces the "too long" time by an order of magnitude, which brings down the failure mode from half-a-work-day to a "run this and grab a meal/coffee" order of time.

If we want to make this configurable, I'd prefer if we do that in a separate change. I think this one is a small independently sensible change that makes a lot of sense in the context of backjumping being introduced, which reduces the number of rounds needed for complex resolves by skipping over irrelevant packages.

maybe pip install --dry-run --report should have a larger number of rounds

I'm wary of having different behaviours for flags that don't say anything about that. If we want this to be changed, then we should make this a knob.

do we need to do anything to avoid the risk of breaking complex installs with the lower limit?

There's a small risk, but I think it's largely mitigated by the fact that the resolver is going to use fewer rounds to get to a relevant package.

To me the only 2 numbers that have a clear understanding of what they mean are 0 and infinity, if infinity is set then Pip should find a solution if one exists (assuming no other limits) and if 0 then no backtracking should be done and the user has to fix their requirements/constraints.

Well... I think that understanding is incorrect.

A round is the resolver attempting to "pin" a single unsatisfied name in the graph. Having max_rounds=0 means that the resolver will be able to resolve at most 0 packages. Backtracking means that if you have 100 names in the final graph, it would take the resolver >100 rounds to figure out a solution. Having max_rounds=infinity means that the resolver will work through as much of the search graph as necessary to get to resolution impossible / resolution found.

[pfmoore]

If we want to make this configurable, I'd prefer if we do that in a separate change.

Fair. And I'm still not convinced this is something we should expect users to be able to configure meaningfully.

I'm wary of having different behaviours for flags that don't say anything about that.

Agreed, that's a reasonable point.

There's a small risk, but I think it's largely mitigated by the fact that the resolver is going to use fewer rounds to get to a relevant package.

Makes sense.

OK, I think the options have been considered, and on reflection I'm happy that we've thought about them but it's not worth doing anything specific. +1 from me on this.

[notatallshaw]

A round is the resolver attempting to "pin" a single unsatisfied name in the graph. Having max_rounds=0 means that the resolver will be able to resolve at most 0 packages.

Does that mean if a user has a frozen set of requirements that needed no backtracking and they had 1 more requirement that max rounds it would fail?

[pradyunsg]

Does that mean if a user has a frozen set of requirements that needed no backtracking and they had 1 more requirement that max rounds it would fail?

Yup. Even equal to max rounds will fail, because it'll have a round of "I need nothing more".

❯ pip --version
pip 23.1.dev0 from /Users/pradyunsg/Developer/github/pip/src/pip (python 3.11)
❯ g diff
diff --git a/src/pip/_internal/resolution/resolvelib/resolver.py b/src/pip/_internal/resolution/resolvelib/resolver.py
index 47bbfecce..ce92a5f6c 100644
--- a/src/pip/_internal/resolution/resolvelib/resolver.py
+++ b/src/pip/_internal/resolution/resolvelib/resolver.py
@@ -88,7 +88,7 @@ def resolve(
         )
 
         try:
-            limit_how_complex_resolution_can_be = 200000
+            limit_how_complex_resolution_can_be = 2
             result = self._result = resolver.resolve(
                 collected.requirements, max_rounds=limit_how_complex_resolution_can_be
             )
❯ pip install sampleproject
Collecting sampleproject
  Using cached sampleproject-3.0.0-py3-none-any.whl (4.7 kB)
Collecting peppercorn (from sampleproject)
  Using cached peppercorn-0.6-py3-none-any.whl (4.8 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/commands/install.py", line 398, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2
❯ pip install sampleproject==3.0.0 peppercorn==0.6  
Collecting sampleproject==3.0.0
  Using cached sampleproject-3.0.0-py3-none-any.whl (4.7 kB)
Collecting peppercorn==0.6
  Using cached peppercorn-0.6-py3-none-any.whl (4.8 kB)
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/commands/install.py", line 398, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
                            ^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pradyunsg/Developer/github/pip/src/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
    raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 2
❯ pip install peppercorn==0.6                     
Collecting peppercorn==0.6
  Using cached peppercorn-0.6-py3-none-any.whl (4.8 kB)
Installing collected packages: peppercorn
Successfully installed peppercorn-0.6

[notatallshaw]

Even though this already got merged I did run some tests over the last day, summary of my findings:

• It's possible to at least contrive requirements that takes more than 200'000 rounds to finish
• In my testing these requirements were considerably more complex than issues posted to Pip's issue tracker
• Each individual round of resolution took much longer, presumably related to the complexity of the requirements, in my examples that pass 200'000 rounds it took over 6 hours on my very fast computer
• In my examples I was able to show they were not resolvable as I took a subset of the requirements and got a resolution impossible error