Improve handling of rustc_private

[DJMcNab]

This PR changes how rust-analyzer handles rustc_private. In particular, packages now must opt-in to using rustc_private in Cargo.toml, by adding:

[package.metadata.rust-analyzer]
rustc_private=true

This means that depending on crates which also use rustc_private will be significantly improved, since their dependencies on the rustc_private crates will be resolved properly.

A similar approach could be used in #6714 to allow annotating that your package uses the test crate, although I have not yet handled that in this PR.

Additionally, we now only index the crates which are transitive dependencies of rustc_driver in the rustcSource directory. This should not cause any change in behaviour when using rustcSource: "discover", as the source used then will only be a partial clone. However, if rustcSource pointing at a local checkout of rustc, this should significantly improve the memory usage and lower indexing time. This is because we avoids indexing all crates in src/tools/, which includes rust-analyzer itself.

Furthermore, we also prefer named dependencies over dependencies from rustcSource. This ensures that feature resolution for crates which are depended on by both rustc and your crate uses the correct set for analysing your crate.

See also introductory zulip stream

I have tested this in priroda, and it provides a significant improvement to the development experience (once I give miri the required data in Cargo.toml)

Todo:

• Documentation

This is ready to review, and I will add documentation if this would be accepted (or if I get time to do so anyway)

[DJMcNab]

I have just discovered that (it appears that) any crate can use extern crate regex_syntax; (for example) if rustc-dev is installed, and just use it directly, without depending on it through cargo. This seems like a non-ideal status quo.

Never mind, we have the rustc_private feature enabled. See rust-lang/rust#50373

[DJMcNab]

Packages which probably currently use rustc_private and rust-analyzer:

• https://github.com/rust-lang/rust-clippy
• https://github.com/oli-obk/priroda/
• https://github.com/EmbarkStudios/rust-gpu
• https://github.com/rust-lang/miri
  If we do got with this, we should have an additional warning when we get 'unresolved extern crate' for crates which start with rustc_.

[bjorn3]

In rustc_codegen_cranelift I don't use rust-analyzer.rustcDev as it uses too much memory. Instead I just disabled the errors for unresolved extern crate.

[DJMcNab]

This PR should also significantly reduce memory usage from using rustcSource @bjorn3.

(This is because we now only index the crates brought in by rustc-dev, instead of all crates in rustcSource).

[bjorn3]

I just tried it out. The memory usage is indeed lower. However it tried to run cargo check on the rustc source.

[DJMcNab]

@bjorn3 I've added a setting which could help - rust-analyzer.cargo.disableRustcBuildScripts.

[matklad]

Let's move this and rustcSource into rustc_private section, so that we have

"rustcPrivate.src": ...,
"rustcPrivate.runBuildScripts": ...,

[DJMcNab]

Would cargo.rustcPrivate.runBuildScripts and cargo.rustcPrivate.src make sense? Since we only have special behaviour for the rustcSource for cargo, afaik.

[matklad]

I'd rathe keep it top level. The fact that it is rustcPrivate is more important than that it is cargo.

[matklad]

but no strong opinion here!

[matklad]

This looks good to me!

I am a bit hesitant about putting rust-analyzer string in Cargo.toml, I think that generally would be a mistake. However, this is rustc_private stuff, so it doesn't really matter.

Hm, now that I've spelled this out, I wonder if we can do somthing worse :D Can we just read lib.rs and see if it contains rustc_private string?

[matklad]

Can we just read lib.rs and see if it contains rustc_private string?

I think we can just bypass VFS here and use just std::fs::read_to_string. That would be bad as a "proper" way to do this, but for rustc private it's better than extending the inteface to allow loading peek into sources.

EDIT: seems to fancy, and without a clear place to report an error, lets keep this as a key in metadata.

[jyn514]

Hm, now that I've spelled this out, I wonder if we can do somthing worse :D Can we just read lib.rs and see if it contains rustc_private string?

This will break if only one crate in the workspace uses rustc_private (unless you mean to do this for each crate?)

[jyn514]

@bjorn3 I've added a setting which could help - rust-analyzer.cargo.disableRustcBuildScripts.

A setting seems kind of weird, cargo check will never work on rustc sources. Could you turn it on unconditionally when package.metadata.rust-analyzer is enabled instead?

[matklad]

bors r+

[bors]

Canceled.

[bors]

🔒 Permission denied

Existing reviewers: click here to make DJMcNab a reviewer

[matklad]

bors r+

[bors]

Build succeeded:

• Rust (macos-latest)
• Rust (ubuntu-latest)
• Rust (windows-latest)
• TypeScript (ubuntu-latest)
• TypeScript (windows-latest)