`rust-analyzer` discoverConfig integration #3073

bobozaur · 2024-12-10T09:53:21Z

Adds a target that can be used for project auto-discovery by using the discoverConfig settings as described in the rust-analyzer user manual.

Unlike the gen_rust_project target, this can be used for dynamic project discovery, and passing {arg} to discoverConfig.command can split big repositories into multiple, smaller workspaces that rust-analyzer switches between as needed. Large repositories can make it OOM.

At amo, we've used a similar implementation for a while with great success, which is why we figured we might upstream it. The changes also include two additional output groups to ensure that proc-macros and build script targets are built, as rust-analyzer depends on these to provide complete IDE support.

Additionally, the PR makes use of the output_base value in bazel invocations. We found it helpful to have tools such as rust-analyzer and clippy run on a separate bazel server than the one used for building. And a config_group argument was added to provide the ability to provide a config group to bazel invocations.

An attempt to get codelens actions to work was done as well, particularly around tests and binaries. They seem to work, but I'm not 100% sure whether the approach taken is the right one.

Closes #2755 .

google-cla · 2024-12-10T09:53:27Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

…dirs output groups

bobozaur · 2024-12-16T14:06:02Z

@sam-mccall I noticed you've been working on things adjacent to this in other recent PRs. Just adding you here for visibility reasons. Any advice you can offer regarding the PR is more than welcome.

sam-mccall · 2024-12-18T23:16:08Z

@sam-mccall I noticed you've been working on things adjacent to this in other recent PRs. Just adding you here for visibility reasons. Any advice you can offer regarding the PR is more than welcome.

Thanks! Yes I had a draft version running locally that I'd planned to clean up and send, but I'm happy you got there first.
I'm not an owner here, but happy to provide feedback and test this out.

This PR combines a few logically-separate changes. I get it - there are a bunch of details that all need to be right for this to work end-to-end. (I ran into some overlapping-but-different set of these myself). I'll try to understand them all and provide high-level comments first, and then let you figure out whether to split them out and land the more "obvious" stuff first. (e.g. having a separate output-base and injecting blaze configuration are useful, but different users will have different needs here).

(I did spot this last friday and started to review, but have been unwell this week...)

sam-mccall

This is great stuff, really thorough job! Some parts of the protocol (logging & runnables) were totally new to me.

High level:

starlark changes (proc macros & build scripts) are good. Related: rust_analyzer: generate more reqired sources #3031 includes some other generated files.
A separate binary for discover sounds good to me (I had this as a flag on gen_rust_project, but it's messy)
this is background infra that should "do what I mean", as such we want fewer flags and more detection.
some things we will eventually want should inform flags/detection:
- become a standalone binary (not invoked through blaze run)
- automatically select an output_base
a BlazeSpec (argv0, workspace, execroot, ...) would be a useful abstraction, replacing Config and long param lists
clearer split between main (config + protocol) and library (build+query+describe) would make the code easier to follow.

Let me know what you think - as I said I'm not an owner here. I can nag one to get you a stamp :-) but also feel free to get a second opinion instead.

If you're interested, this could be a few PRs - might go quicker, but totally up to you:

proc macros + build scripts
refactor current bin+lib to make lib reusable
add discover tool
runnables

sam-mccall · 2024-12-19T13:19:14Z

rust/private/rust_analyzer.bzl

        build_info = build_info,
    ))

    return [
        rust_analyzer_info,
-        OutputGroupInfo(rust_analyzer_crate_spec = rust_analyzer_info.crate_specs),
+        OutputGroupInfo(


I think rust_analyzer_build_info_out_dirs is really about ensuring generated sources are available, and should be named as such: rust_analyzer_srcs. See https://github.com/bazelbuild/rules_rust/pull/3031/files. These don't need to be distinct output groups as from the caller's point of view it doesn't make sense to want one without the other.

Having three output groups does make sense to me though: crate specs are the bare minimum and I can imagine producing the rest asynchronously, and proc macros are optional (we use a toolchain that doesn't even have the proc-macro-srv...)

sam-mccall · 2024-12-19T13:28:45Z

rust/private/providers.bzl

        "deps": "List[String]: IDs of direct dependency crates",
        "env": "Dict[String: String]: Environment variables, used for the `env!` macro",
        "id": "String: Arbitrary unique ID for this crate",
-        "proc_macro_dylib_path": "File: compiled shared library output of proc-macro rule",
+        "proc_macro_dylib": "File: compiled shared library output of proc-macro rule",


Nit: "if this is a proc-macro target, the shared-library output"?

(wasn't clear to me which macro, and "rule" is wrong here I think)

sam-mccall · 2024-12-19T13:30:38Z

rust/private/providers.bzl

@@ -162,17 +162,21 @@ RustAnalyzerInfo = provider(
        "cfgs": "List[String]: features or other compilation `--cfg` settings",
        "crate": "CrateInfo: Crate information.",
        "crate_specs": "Depset[File]: transitive closure of OutputGroupInfo files",
+        "proc_macro_dylibs": "Depset[File]: transitive closure of OutputGroupInfo files",


These should have clear names and comments (the comment on crate_specs is bad; maybe fix while here?)

sam-mccall · 2024-12-19T13:32:03Z

rust/private/rust_analyzer.bzl

@@ -45,6 +45,8 @@ def write_rust_analyzer_spec_file(ctx, attrs, owner, base_info):
        RustAnalyzerInfo: Info with the embedded spec file.
    """
    crate_spec = ctx.actions.declare_file("{}.rust_analyzer_crate_spec.json".format(owner.name))
+    proc_macro_dylibs = [base_info.proc_macro_dylib] if base_info.proc_macro_dylib else None


this function is just to write the file, everything with the RustAnalyzerInfo should be a pure passthrough.

(This function is ugly and there's probably a way to get rid of it, but failing that it should at least stay trivial)

sam-mccall · 2024-12-19T13:33:32Z

rust/private/rust_analyzer.bzl

@@ -221,6 +247,13 @@ def _create_single_crate(ctx, attrs, info):
    crate["root_module"] = path_prefix + info.crate.root.path
    crate["source"] = {"exclude_dirs": [], "include_dirs": []}

+    # Store build system related info only for local crates


this comment echoes the code, can you say why instead?

To me it's not obvious why, but I guess it's to prevent blaze test runnables from showing up for other crates? But is "local" the right choice there, rather than "top-level" i.e. targets from the selected blaze package?

sam-mccall · 2024-12-19T15:59:41Z

tools/rust_analyzer/rust_project.rs

@@ -108,6 +223,31 @@ pub fn generate_rust_project(
        sysroot: Some(sysroot.into()),
        sysroot_src: Some(sysroot_src.into()),
        crates: Vec::new(),
+        runnables: vec![


(no action needed) runnables is not documented in the rust-project.json spec, we should fix that.

sam-mccall · 2024-12-19T16:05:48Z

tools/rust_analyzer/rust_project.rs

@@ -108,6 +223,31 @@ pub fn generate_rust_project(
        sysroot: Some(sysroot.into()),
        sysroot_src: Some(sysroot_src.into()),
        crates: Vec::new(),
+        runnables: vec![
+            Runnable {


What's the intended use of build?

If it's to get diagnostics (which RunnableKind::Check suggests), it's not clear to me if we should pass --config, --output_base etc as we're a tool, or not pass them as we're acting as a simple proxy for the user.

I'm fine with either answer but let's say why in a comment.

Part of this is: are we expecting --config to be something like --config=generate_simplified_sources_for_rust_analyzer (should be ignored) or --config=macos (should be passed)?

sam-mccall · 2024-12-19T16:24:34Z

tools/rust_analyzer/rust_project.rs

-        .replace("__OUTPUT_BASE__", output_base)
-        .replace("__WORKSPACE__", workspace);
+    let rust_project_content = serde_json::to_string_pretty(rust_project)?;
+    let rust_project_content = normalize_project_string(


the layering is unclear: here you're calling normalize_project_string within the rust_project module, but in the other case (discover) the caller is expected to invoke it.

This normalization is messy :-(
As-is, I think the cleanest solution is to consistently call the normalize function from the top level, right before doing the IO (and not have the IO buried inside libraries).

sam-mccall · 2024-12-19T16:27:35Z

tools/rust_analyzer/lib.rs

        .env_remove("BAZELISK_SKIP_WRAPPER")
        .env_remove("BUILD_WORKING_DIRECTORY")
        .env_remove("BUILD_WORKSPACE_DIRECTORY")
+        .arg(format!("--output_base={output_base}"))
        .arg("build")


(no action needed) When we're running in the background, we want to be as forgiving as possible if the user's current state is broken. So eventually we should have --nocheck_visibility here and possibly elsewhere.

sam-mccall · 2024-12-19T16:33:08Z

tools/rust_analyzer/lib.rs

@@ -1,62 +1,153 @@
+mod aquery;


In general the layering/responsibility split between main, this lib, and its submodules feels pretty unclear. (This is true before your changes, but it's more load-bearing after them!)

I'd suggest main should be responsible for flag parsing, protocol (serialization, deserialization, io), and most things that are inherently tool-specific. I don't see a clear reason that we want discover_rust_project and write_rust_project as different library entry points.

bobozaur · 2024-12-23T10:51:48Z

@sam-mccall Sorry for the delay and thank you for the thorough review! I'm not going to have access to my laptop for a few more days but I plan to address all of this after Christmas. Happy holidays :)!

bobozaur force-pushed the rust-analyzer-discover-config branch 4 times, most recently from d8ef7e4 to 80a45c1 Compare December 10, 2024 17:50

bobozaur added 6 commits December 13, 2024 10:19

added usage of camino crate

8e51c2d

add discover_rust_project target

dc3ea50

fix tests

3806c52

allow passing in a config group

683440e

remove targets query

5ddbec5

add rust_analyzer_proc_macro_dylibs and rust_analyzer_build_info_out_…

ea06599

…dirs output groups

bobozaur force-pushed the rust-analyzer-discover-config branch from 267aa69 to ea06599 Compare December 13, 2024 09:19

bobozaur added 2 commits December 16, 2024 15:55

improve crate spec merging

ac1dbe0

add is_test to crate spec

5de25c8

better runnables support

98aa3b9

sam-mccall reviewed Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`rust-analyzer` discoverConfig integration #3073

`rust-analyzer` discoverConfig integration #3073

bobozaur commented Dec 10, 2024 •

edited

Loading

google-cla bot commented Dec 10, 2024

bobozaur commented Dec 16, 2024

sam-mccall commented Dec 18, 2024

sam-mccall left a comment

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

sam-mccall Dec 19, 2024

bobozaur commented Dec 23, 2024

rust-analyzer discoverConfig integration #3073

Are you sure you want to change the base?

rust-analyzer discoverConfig integration #3073

Conversation

bobozaur commented Dec 10, 2024 • edited Loading

google-cla bot commented Dec 10, 2024

bobozaur commented Dec 16, 2024

sam-mccall commented Dec 18, 2024

sam-mccall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bobozaur commented Dec 23, 2024

`rust-analyzer` discoverConfig integration #3073

`rust-analyzer` discoverConfig integration #3073

bobozaur commented Dec 10, 2024 •

edited

Loading