Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caching the dependencies #6853

Merged
merged 10 commits into from
Apr 26, 2019
Merged

Caching the dependencies #6853

merged 10 commits into from
Apr 26, 2019

Conversation

Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Apr 15, 2019

There are 2 sources of facts for the resolver:

  1. The Registry tells us for a Dependency what versions are available to fulfil it.
  2. The Summary tells us for a version (and features) what dependencies need to be fulfilled for it to be activated.

The Registry was cached with a RegistryQueryer back in #5112. This adds a DepsCache to cache the calculation of which dependencies are activated by features.

In the happy path flag_activated means that we don't get to reuse build_deps, but the more we backtrack the more time we save. In pathological cases like #6258 (comment), I have measured this as a 10% improvement with release.

This also means that build_deps can be run in a context free way, which may be useful in a follow up PR to solve #6258 (comment).

@rust-highfive
Copy link

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 15, 2019
@nrc
Copy link
Member

nrc commented Apr 15, 2019

r? @alexcrichton

@bors
Copy link
Contributor

bors commented Apr 18, 2019

☔ The latest upstream changes (presumably #6840) made this pull request unmergeable. Please resolve the merge conflicts.

Copy link
Member

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks pretty reasonable to me, thanks for separating the commits! I think the organization between DepCache and RegistryQueryer may want to be tweaked slightly, but it's not the end of the world either way.

The main thing I think this needs is to expand the documentation as to why this caching strategy is valid, aka just some documentation on DepsCache as to what it's caching and why it's possible.

src/cargo/core/resolver/context.rs Outdated Show resolved Hide resolved
@@ -89,26 +89,26 @@ impl ResolverProgress {
}
}

#[derive(Clone, Copy)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like it's somewhat of a loss, but is there a reason we couldn't use &'a BTreeSet here for the Required list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see how. (love to be proven wrong.) This change was to get rid of the lifetime, so that it could be used as the key in the DepsCache above. Then I switched from a Vec to a BTreeSet to reduce the number of equivalent configurations that can end up in the cache. Then added the Rc as that was the kind of reference I usually had.

This explanation should be a comment. I will add after #6860 lands.

src/cargo/core/resolver/dep_cache.rs Outdated Show resolved Hide resolved
@@ -167,6 +167,36 @@ impl<'a> RegistryQueryer<'a> {
}
}

pub fn build_deps(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to below, could this document the return value and what each value of the tuple is?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(also while you're at it if you can add general documentation for what this method is doing that'd be great!

src/cargo/core/resolver/dep_cache.rs Outdated Show resolved Hide resolved
candidate: &Summary,
method: &Method,
) -> ActivateResult<Rc<(Vec<String>, HashSet<InternedString>, Rc<Vec<DepInfo>>)>> {
if let Some(out) = self
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could a comment be added here as to why we can cache the results? It'd be good to explain why this cache works the way it does and what enables it (e.g. the things that don't change and how this is a "pure" query which can be memoized.

@bors
Copy link
Contributor

bors commented Apr 25, 2019

☔ The latest upstream changes (presumably #6860) made this pull request unmergeable. Please resolve the merge conflicts.

@Eh2406
Copy link
Contributor Author

Eh2406 commented Apr 25, 2019

How does this look now?

@alexcrichton
Copy link
Member

@bors: r+

Looks great to me, thanks for the updates!

@bors
Copy link
Contributor

bors commented Apr 26, 2019

📌 Commit 79714ba has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Apr 26, 2019
@bors
Copy link
Contributor

bors commented Apr 26, 2019

⌛ Testing commit 79714ba with merge 0c891a0...

bors added a commit that referenced this pull request Apr 26, 2019
Caching the dependencies

There are 2 sources of facts for the resolver:
1. The `Registry` tells us for a Dependency what versions are available to fulfil it.
2. The `Summary` tells us for a version (and features) what dependencies need to be fulfilled for it to be activated.

The `Registry` was cached with a `RegistryQueryer` back in #5112. This adds a `DepsCache` to cache the calculation of which dependencies are activated by features.

In the happy path `flag_activated` means that we don't get to reuse `build_deps`, but the more we backtrack the more time we save. In pathological cases like #6258 (comment), I have measured this as a 10% improvement with release.

This also means that `build_deps` can be run in a context free way, which may be useful in a follow up PR to solve #6258 (comment).
@bors
Copy link
Contributor

bors commented Apr 26, 2019

☀️ Test successful - checks-travis, status-appveyor
Approved by: alexcrichton
Pushing 0c891a0 to master...

@bors bors merged commit 79714ba into rust-lang:master Apr 26, 2019
@Eh2406 Eh2406 deleted the caching-the-dependency branch April 30, 2019 02:07
@ehuss ehuss added this to the 1.36.0 milestone Feb 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants