Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: hybrid incremental solver and performance improvements #349

Merged
merged 22 commits into from
Sep 25, 2023

Conversation

baszalmstra
Copy link
Collaborator

@baszalmstra baszalmstra commented Sep 25, 2023

This is a big PR that does two things:

Incremental solver

The solver is now incremental by default. This means that the solver will only request information from the DependencyProvider when it is relatively sure that that information is needed to come up with a solution. Previously the solver would greedily request all information about the problem space up front. This made the solver very unsuitable for cases where requesting information about packages is expensive.

Optionally a DependencyProvider has the ability to hint to the solver that it already has dependency information available for some candidates. The solver will use this information to increase its knowledge of the problem space without requiring network operations. The latter is used for the conda solver.

In general, it's better to have all the information available upfront because if the solver has "all" the information available it is able to reach a solution quicker because it doesn't have to guess about information it doesn't know.

Performance improvements

This PR also implements a performance improvement by being more thoughtful about making decisions about the next steps to take in the algorithm. Instead of picking the first available decision it can make it checks all possible choices to make and picks the decision that involves the least amount of versions to pick.

This is similar to what pubgrub does:

https://github.com/dart-lang/pub/blob/master/doc/solver.md#decision-making

In the future, we might take a look at different heuristics but this implementation is good enough for now.

Some results:

libsolv libsolv-rs 0.7.0 libsolv-rs (this)
python=3.9 7.03ms 3.47ms 3.57ms
xtensor, xsimd 5.28ms 2.00ms 2.22ms
tensorflow 773.27ms 407.98ms 152.94ms
quetz 1380.8ms 1684.7ms 301.53ms
tensorboard=2.1.1, grpc-cpp=1.39.1 515.25ms 122.48ms 89.41ms
rubin-env 7459.7ms 3633.2ms 1798.8ms

Note that for "simple" cases this PR is slower but just slightly and for "complex" cases this PR is much faster.

@baszalmstra baszalmstra merged commit 61ed341 into conda:main Sep 25, 2023
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants