Turn crate store into a resolver output #65625

petrochenkov · 2019-10-20T14:09:58Z

Crate store (CStore) is a vector of data (CrateMetadata) associated with extern crates loaded during the current compilation session.

All crates are loaded in the resolver when resolving either paths pointing to extern prelude or extern crate items. (There are also a couple of crates like panic runtime that are loaded kind of like implicit extern crates, but that also happens in resolve.)

The use of CStore from rustc_plugin (which is outside of the resolver) was unnecessary because legacy plugins are not added to the crate store and don't use CrateNums.

So, CStore can be produced by the resolver instead of being kept in some really global data (rustc_interface::Compiler) like now.

As a result of crate store being more "local" we can now remove some locks and Lrcs.

petrochenkov · 2019-10-20T14:12:05Z

src/librustc_metadata/cstore.rs

 pub struct CStore {
-    metas: RwLock<IndexVec<CrateNum, Option<Lrc<CrateMetadata>>>>,
-    crate metadata_loader: Box<dyn MetadataLoader + Sync>,
+    metas: IndexVec<CrateNum, Option<Lrc<CrateMetadata>>>,


Unfortunately we cannot remove this Lrc because CStore has to be cloneable.
EDIT: The reason - #65625 (comment).

Wow, I didn't even realize that was a thing. Do you know where the clones are coming from?

I guess we could get rid of it by having the expansion query produce the CStore structure and redirect later uses to that instead. That would make the structure more like before this PR. It doesn't seem like a huge win right now though. It might be worth waiting on moving things into TyCtxt. Maybe we'd replace it by a crate_metadata(CrateNum) -> &'tcx CrateMetadata query or something.

Yeah, this certainly can be done later if "later" is a better time due to other changes (like TyCtxt).
The clone itself (the primary effect) is cheap - only a vector of Lrcs is cloned.
The locks around CrateMetadata::{dependencies,dep_kind,extern_crate} (the secondary effect) are also unlikely to be accessed often and cause performance issues.

petrochenkov · 2019-10-20T14:16:57Z

Regarding locks in CrateMetadata:

source_map_import_info and dep_node_index are created lazily on the first use, so they need synchronization, but can probably use some of the Once primitives instead.
dependencies, dep_kind and extern_crate are only under locks because CrateMetadata is under Lrc (Turn crate store into a resolver output #65625 (comment)), they are used only from inside the crate loader and could just use mutable references otherwise.

petrochenkov · 2019-10-20T14:21:24Z

src/librustc_interface/passes.rs

@@ -211,7 +171,7 @@ impl BoxedResolver {
            Err(resolver) => {
                let resolver = &*resolver;
                resolver.borrow_mut().access(|resolver| {
-                    ExpansionResult::from_resolver_ref(resolver)
+                    ExpansionResult::from_resolver_outputs(resolver.clone_outputs())


This is why we can't have nice things.
This case puts the cloneability requirement on resolver outputs (https://github.com/rust-lang/rust/pull/65625/files#r336780545), but it actually never happens in rustc, only in rustdoc and perhaps other tools using rustc_interface.

Why would cloning be needed? Is this that support for resolving things after rustc_resolve finishes?
In that case I'm not sure why we need to clone the outputs - is it to be able to keep the original resolver around?

Not sure, rustc_interface is very hard to reason about. ¯_(ツ)_/¯
It needs some investigation, but I'm not sure I'll get to it until the next weekend.

This clone only happens with rustdoc since it needs the resolver available at a later stage. It would be nicer if we could extract the information rustdoc needs instead.

Can we split the resolver up so that we can pass a &CStore later to the parts of it that we need to keep around? (i.e. the local module tree)

We could make expansion query produce the final CStore structure and I said in my other comment. That won't help to get rid of the clone here though.

I'd prefer if rustdoc didn't cause that clone to happen but instead did its resolving in some other way.

It really feels like the new code is not worse, right? So we can definitely leave this for a future PR (filing an issue wouldn't hurt). I'd rather not block this PR on that cleanup.

This clone was there before this PR.

petrochenkov · 2019-10-20T14:22:17Z

cc @rust-lang/compiler @Mark-Simulacrum
Several people may be interested in this PR because it touches rustc_interface and removes locks.

petrochenkov · 2019-10-20T14:45:55Z

r? @Mark-Simulacrum or @Zoxc probably

Mark-Simulacrum

Overall looks great, excellent cleanup!

I agree with @eddyb that the cloning is unfortunate but ultimately seems 'fine' for now, we can clean it up in a future PR.

Mark-Simulacrum · 2019-10-20T17:08:26Z

src/librustc/middle/cstore.rs

@@ -191,6 +191,8 @@ pub trait MetadataLoader {
                          -> Result<MetadataRef, String>;
 }

+pub type MetadataLoaderDyn = dyn MetadataLoader + Sync;


Do we need this to be Sync even in the non-parallel compiler case? Maybe we want sync::Sync here which is either the proper marker trait from core or the data structures one which is just auto for everything?

The metadata loaders have no fields, so it doesn't matter which Sync trait is used.

Mark-Simulacrum · 2019-10-20T17:14:21Z

src/librustc_interface/passes.rs

@@ -159,13 +156,12 @@ pub fn configure_and_expand(
 }

 pub struct ExpansionResult {


Is there a reason to re-wrap the resolve outputs here? I guess we do get Steal...

ExpansionResult is removed.

Mark-Simulacrum · 2019-10-20T17:30:02Z

r=me unless you want to investigate my comments in this PR

Zoxc · 2019-10-20T21:49:29Z

This PR looks good to me too.

petrochenkov · 2019-10-21T21:12:31Z

Let's run perf on this.
@bors try @rust-timer queue

rust-timer · 2019-10-21T21:12:32Z

Awaiting bors try build completion

bors · 2019-10-21T21:12:41Z

⌛ Trying commit 2a329d24c0f81e3485ab3148ea4be60e3beea33b with merge fb82105b083af835f36cba113878eabd97ab8573...

bors · 2019-10-22T00:15:04Z

☀️ Try build successful - checks-azure
Build commit: fb82105b083af835f36cba113878eabd97ab8573 (fb82105b083af835f36cba113878eabd97ab8573)

rust-timer · 2019-10-22T00:15:05Z

Queued fb82105b083af835f36cba113878eabd97ab8573 with parent 10f12fe, future comparison URL.

petrochenkov · 2019-10-22T05:10:02Z

Let's run perf on this.

Not sure why I did that, there should be ~no changes for the non-parallel case.
@bors r=Mark-Simulacrum,Zoxc

bors · 2019-10-22T05:10:03Z

📌 Commit 2a329d24c0f81e3485ab3148ea4be60e3beea33b has been approved by Mark-Simulacrum,Zoxc

rust-timer · 2019-10-22T06:47:27Z

Finished benchmarking try commit fb82105b083af835f36cba113878eabd97ab8573, comparison URL.

bors · 2019-10-24T07:18:53Z

☔ The latest upstream changes (presumably #65733) made this pull request unmergeable. Please resolve the merge conflicts.

Plugin search doesn't need a crate loader, only crate locator

Crate metadatas are still stored as `Lrc<CrateMetadata>` in `CStore` because crate store has to be cloneable due to `Resolver::clone_outputs`.

petrochenkov · 2019-10-24T18:13:12Z

@bors r=Mark-Simulacrum,Zoxc

bors · 2019-10-24T18:13:13Z

📌 Commit 94216ce has been approved by Mark-Simulacrum,Zoxc

bors · 2019-10-24T18:13:13Z

🌲 The tree is currently closed for pull requests below priority 1000, this pull request will be tested once the tree is reopened

…rum,Zoxc Turn crate store into a resolver output Crate store (`CStore`) is a vector of data (`CrateMetadata`) associated with extern crates loaded during the current compilation session. All crates are loaded in the resolver when resolving either paths pointing to extern prelude or `extern crate` items. (There are also a couple of crates like panic runtime that are loaded kind of like implicit `extern crate`s, but that also happens in resolve.) The use of `CStore` from `rustc_plugin` (which is outside of the resolver) was unnecessary because legacy plugins are not added to the crate store and don't use `CrateNum`s. So, `CStore` can be produced by the resolver instead of being kept in some really global data (`rustc_interface::Compiler`) like now. As a result of crate store being more "local" we can now remove some locks and `Lrc`s.

@ghost

Rollup of 8 pull requests Successful merges: - #65625 (Turn crate store into a resolver output) - #65627 (Forbid non-`structural_match` types in const generics) - #65710 (Update cargo) - #65729 (Update test cases for vxWorks) - #65746 (Tweak format string error to point at arguments always) - #65753 (Don't assert for different instance on impl trait alias) - #65755 (Avoid ICE when adjusting bad self ty) - #65766 (Update hashbrown to 0.6.2) Failed merges: r? @ghost

bors · 2019-10-25T04:16:54Z

☔ The latest upstream changes (presumably #65771) made this pull request unmergeable. Please resolve the merge conflicts.

rust-highfive assigned matthewjasper Oct 20, 2019

petrochenkov commented Oct 20, 2019

View reviewed changes

rust-highfive assigned Mark-Simulacrum and unassigned matthewjasper Oct 20, 2019

petrochenkov added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 20, 2019

Mark-Simulacrum approved these changes Oct 20, 2019

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 21, 2019

petrochenkov force-pushed the cstore branch from 71bf704 to 2a329d2 Compare October 21, 2019 21:11

petrochenkov added S-waiting-on-perf Status: Waiting on a perf run to be completed. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 21, 2019

bors added the S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. label Oct 22, 2019

petrochenkov removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Oct 22, 2019

Centril mentioned this pull request Oct 23, 2019

Rollup of 9 pull requests #65723

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 24, 2019

petrochenkov added 10 commits October 24, 2019 20:50

rustc_driver: Remove unnecessary use of crate store

2c16f84

rustc_metadata: Remove unnecessary use of crate store in plugin loader

2cda75c

rustc_metadata: Move some code around

175d325

Plugin search doesn't need a crate loader, only crate locator

rustc: Add a convenience alias for dyn MetadataLoader + Sync

222503a

resolve: Privatize all resolver fields

fb353f0

rustc: Combine resolver outputs into a single struct

5fd796a

Turn crate store into a resolver output

3534ca8

rustc_metadata: Remove RwLock from CStore

c5fee33

rustc_metadata: Minimize use of Lrc in crate store

9f5a530

Crate metadatas are still stored as `Lrc<CrateMetadata>` in `CStore` because crate store has to be cloneable due to `Resolver::clone_outputs`.

rustc_interface: Remove ExpansionResult and some Steals

94216ce

petrochenkov force-pushed the cstore branch from 2a329d2 to 94216ce Compare October 24, 2019 18:12

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 24, 2019

Centril mentioned this pull request Oct 24, 2019

Rollup of 8 pull requests #65771

Merged

bors merged commit 94216ce into rust-lang:master Oct 25, 2019

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Oct 25, 2019

petrochenkov mentioned this pull request Jan 26, 2020

rustdoc only: duplicate lang item panic_halt #68427

Closed

petrochenkov mentioned this pull request Feb 25, 2020

Properly handle Spans that reference imported SourceFiles #68941

Merged

jyn514 mentioned this pull request Apr 1, 2021

rustdoc: Stop cloning the resolver #83761

Closed

6 tasks

		@@ -159,13 +156,12 @@ pub fn configure_and_expand(
		}

		pub struct ExpansionResult {

Turn crate store into a resolver output #65625

Turn crate store into a resolver output #65625

Conversation

petrochenkov commented Oct 20, 2019

petrochenkov Oct 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov commented Oct 20, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrochenkov commented Oct 20, 2019

petrochenkov commented Oct 20, 2019

Mark-Simulacrum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mark-Simulacrum commented Oct 20, 2019

Zoxc commented Oct 20, 2019

petrochenkov commented Oct 21, 2019

rust-timer commented Oct 21, 2019

bors commented Oct 21, 2019

bors commented Oct 22, 2019

rust-timer commented Oct 22, 2019

petrochenkov commented Oct 22, 2019

bors commented Oct 22, 2019

rust-timer commented Oct 22, 2019

bors commented Oct 24, 2019

petrochenkov commented Oct 24, 2019

bors commented Oct 24, 2019

bors commented Oct 24, 2019

bors commented Oct 25, 2019

petrochenkov Oct 20, 2019 •

edited

Loading