Rust Analyzer: Generate rust-project.json without explicit target list #907

djmarcin · 2021-08-23T07:31:09Z

This PR refactors the generation of rust-project.json significantly. It's not fully ready to merge and I need some advice on how to test this new design. (@hlopko)

In broad strokes:

The aspect now emits a JSON representation of each crate in the dependency graph
Crate deduplication has moved into Rust because we no longer aggregate crates in starlark
The logic formerly in the rust_analyzer rule has been moved into gen_rust_project

gen_rust_project now manages the parsing of individual crate specs and creating the rust-project.json file. The basic logic is:

Call bazel build using the aspect and a list of targets.
Call bazel aquery using the list of targets to retrieve the list of generated spec files
Parse the individual spec files
Iterate through the list of crates, merging crates into rust-project.json if all dependencies have already been merged into rust-project.json (required due to the self-referential array data structure that rust-project.json uses).
Write the final data structure to rust-project.json in the workspace root.

Compatibility with the existing rust_analyzer rule has been dropped, save for an empty implementation to prevent immediate breakages upon updating rules_rust. In the common integration case where a user just used the defaults, no changes are required and the rust_analyzer rule can be deleted at their convenience.

Unfortunately, the existing tests rely on the rust_analyzer rule, so they are broken by this PR. Additionally, I don't know if this PR can be tested inside bazel because it repeatedly invokes bazel while running. A proper test given this approach would probably require some additional flags, so I'd like to get some feedback before I go down a rabbit hole adding hooks for testing that might end up not being what folks want to see.

/cc @hlopko @UebelAndre

UebelAndre

This looks awesome! You're an absolute legend!!! 😄

I think testing for this one is one would make sense to live more in Rust than in Bazel? I'd expect to see some tests for the functions in gen_rust_project_lib and probably have some rule which explicitly generates a .rust_analyzer_sysroot_src file from the aspect use in the rust_test target for schema/deserialization validation. I feel like that'd provide sufficient coverage for the functionality here?

For testing the starlark code though, I did write some unit tests for Clippy which were the first I'd ever seen written for aspects. Hopefully they can serve as some baseline? I'll keep peeking at this throughout the day and see if I have any new ideas.

UebelAndre · 2021-08-23T14:53:35Z

rust/private/rust_analyzer.bzl

+
+    return [OutputGroupInfo(rust_analyzer_sysroot_src = depset([sysroot_src_file]))]
+
+rust_analyzer_detect_sysroot = rule(


Just so I know I'm on the same page, can you describe what you mean by sysroot for this rule and for the output group field?

The name comes from the field in the rust-project.json, but essentially it's the root under which all the rust source can be located (e.g. std::, core::, test::, etc). I'll add a comment.

hlopko · 2021-08-24T21:15:48Z

I'm super interested in this work, but I'm also on vacation until 6th. I'll try to find time on evenings, but I apologize in advance for slowness. Thank you!

djmarcin · 2021-08-30T07:41:45Z

@hlopko No rush, I’m on vacation this week.

I am particularly curious if you (or anyone else) has immediate concerns about this breaking backward compatibility with the rule? Since it’s an experimental feature I’m hoping that we can just give migration instructions and not worry about attempting to interpret the existing rule. I think it’s probably possible to use the existing rule to override the default --targets if it’s important, though.

UebelAndre · 2021-09-08T15:23:12Z

Hello friends, any updates here? I think at this point everyone is back from vacation but no worries if I miscalculated 😅.

hlopko

I like the approach! Great work!

Correct me if I'm wrong, but this could be trivially scaled down to analyzing a single file (for example currently opened file in the editor). The gen_rust_project can compute the target of the file, and then generate rust-project.json from that.

Well that's not that exciting you may say, and you would be right :)

But imagine rust-analyzer itself detected that the project is using Bazel. It could use the gen_rust_project tool repeatedly for each opened file to fetch all the information it needs, invalidating what needs to be invalidated on changes to BUILD files and bzl files.

IMHO it's worth a shot talking to rust-analyzer folks once this PR lands.

Anyway, this is a great work and I'd be happy to have it merged.

hlopko · 2021-09-09T09:48:16Z

tools/rust_analyzer/aquery.rs

+        .output()?;
+
+    let crate_spec_files =
+        parse_aquery_output_files(execution_root, String::from_utf8(aquery_output.stdout)?)?;


Can we parse directly from the stream, without allocating the string?

hlopko · 2021-09-09T09:51:01Z

tools/rust_analyzer/aquery.rs

+    pub edition: String,
+    pub root_module: String,
+    pub is_workspace_member: bool,
+    pub deps: Vec<String>,


Would a set perform better when merging duplicates below? Would you consider it an overkill to use union find? :)

hlopko · 2021-09-09T10:48:21Z

rust/private/rust_analyzer.bzl

    },
 )

 def _rust_analyzer_aspect_impl(target, ctx):
+    if OutputGroupInfo in target:


Why do we have to "re-provide" output groups here?

This is a little bit of a hack to ensure that the rust_analyzer aspect generates the rust_analyzer_sysroot_src, which doesn't have crate_info so will be skipped by the check on L46. We might be able to use a separate aspect for it though (I'd have to double check if that would be too much duplication).

More generally, we need to re-provide the output groups because otherwise if you did something like bazel build --aspects=@rules_rust//rust:defs.bzl%rust_analyzer_aspect --output_groups=rust_analyzer_crate_spec //path/to/single:target then only the crate_spec file for //path/to/single:target will be generated, but none of its deps. Re-providing OutputGroupInfo ensures that all the dependent crate_spec files are generated as well.

djmarcin

Trying to find some time to circle back on these comments and tests sometime this week.

djmarcin · 2021-09-13T16:31:31Z

rust/private/rust_analyzer.bzl

    },
 )

 def _rust_analyzer_aspect_impl(target, ctx):
+    if OutputGroupInfo in target:


This is a little bit of a hack to ensure that the rust_analyzer aspect generates the rust_analyzer_sysroot_src, which doesn't have crate_info so will be skipped by the check on L46. We might be able to use a separate aspect for it though (I'd have to double check if that would be too much duplication).

More generally, we need to re-provide the output groups because otherwise if you did something like bazel build --aspects=@rules_rust//rust:defs.bzl%rust_analyzer_aspect --output_groups=rust_analyzer_crate_spec //path/to/single:target then only the crate_spec file for //path/to/single:target will be generated, but none of its deps. Re-providing OutputGroupInfo ensures that all the dependent crate_spec files are generated as well.

UebelAndre · 2021-09-24T16:34:55Z

Hello friends, friendly ping here 😅 This feature would be awesome!

purkhusid · 2021-10-04T14:16:00Z

@djmarcin Does this improve the performance of the analyzer rules as well?

hlopko · 2021-10-04T15:03:24Z

@purkhusid is the current performance problematic? Could you file an issue for that with more information?

purkhusid · 2021-10-04T17:10:57Z

@hlopko I created an issue here: #962

hlopko · 2021-10-08T09:18:36Z

Thank you! Yeah this PR may be an (constant-factor) improvement. We'll need to measure/profile. But this PR doesn't change the fact that we have to visit all transitive crates...

UebelAndre · 2021-10-09T01:47:23Z

Is there anything I can do to help drive this PR forward? What are the remaining steps?

djmarcin · 2021-10-26T07:09:56Z

Sorry for letting this sit, I've been super busy and haven't had a chance to revisit. I think mainly the issues are just around writing the tests. Otherwise, we've been using this internally for a while and it seems to be very stable for us.

It looks like perhaps recursively calling bazel isn't the problem I thought it would be, because it seems that https://github.com/bazelbuild/rules_rust/blob/main/test/rustfmt/rustfmt_failure_test.sh does it. I'll use that as an example and try to write a few tests.

UebelAndre · 2021-10-26T16:01:43Z

It looks like perhaps recursively calling bazel isn't the problem I thought it would be, because it seems that https://github.com/bazelbuild/rules_rust/blob/main/test/rustfmt/rustfmt_failure_test.sh does it. I'll use that as an example and try to write a few tests.

Yeah, I think that pattern would be just fine, though I think unit testing could probably solve for identifying subtle regressions. If the calls to bazel were to be moved into their own functions where that's virtually the only thing they do, then I think a lot could be done by mocking what the outputs would be (particularly since I doubt aquery outputs are going to change anytime soon). Tests that actually invoke Bazel seem more integration tests to me but the only thing they could offer that unittesting does not is testing interactions with various Bazel binaries. I leave it up to you and am still happy to do whatever I can to help get this through. I'm really excited about this PR 😄

UebelAndre · 2021-11-05T14:35:01Z

rust/defs.bzl

@@ -107,6 +108,9 @@ rust_common = _rust_common
 rust_analyzer_aspect = _rust_analyzer_aspect
 # See @rules_rust//rust/private:rust_analyzer.bzl for a complete description.

+rust_analyzer_detect_sysroot = _rust_analyzer_detect_sysroot


Does this rule need to be public?

UebelAndre · 2021-11-17T20:52:48Z

These changes were merged in #1010

djmarcin · 2021-11-17T21:42:04Z

Thanks @UebelAndre !

djmarcin added 13 commits August 20, 2021 16:49

Emit rust_analyzer_crate_spec files for all workspace crates

eed713c

WIP

8411612

Generate crate_spec files and read aquery output

ab25cda

Parse crate specs

5134438

Generate rust-project.json file with serde

6b77b17

Handle duplicated crates

531dcad

Generate sysroot_src into a file

0330ade

Clean up & refactoriong

0925a93

Clean up old rules

efaed3c

buildifier

e8fd833

rustfmt

2a2f2ef

Regenerate documentation

aee7472

Remove transitive_deps

9662fa6

google-cla bot added the cla: yes label Aug 23, 2021

UebelAndre requested changes Aug 23, 2021

View reviewed changes

hlopko self-requested a review August 24, 2021 21:14

UebelAndre linked an issue Aug 25, 2021 that may be closed by this pull request

Visibility suggestions for rust_analyzer? #905

Closed

UebelAndre mentioned this pull request Aug 25, 2021

Visibility suggestions for rust_analyzer? #905

Closed

sayrer mentioned this pull request Aug 29, 2021

Enable rust_analyzer and VSCode integration. google/cargo-raze#444

Closed

djmarcin added 3 commits September 1, 2021 22:38

Expand locations in rustc_env variables

b155c65

Clean up & simplify code

bb74ba9

Add basic test for converting Vec<CrateSpec> to RustProject

eccdc8d

djmarcin force-pushed the rust-analyzer branch from b67269c to eccdc8d Compare September 2, 2021 07:10

djmarcin added 2 commits September 2, 2021 22:57

buildifier

174dbc4

Merge remote-tracking branch 'upstream/main' into rust-analyzer

d06225d

hlopko reviewed Sep 9, 2021

View reviewed changes

djmarcin commented Sep 13, 2021

View reviewed changes

hlopko mentioned this pull request Oct 11, 2021

Performance of rust_analyzer #962

Closed

UebelAndre requested changes Nov 5, 2021

View reviewed changes

UebelAndre mentioned this pull request Nov 11, 2021

Rust Analyzer: Generate rust-project.json without explicit target list #1010

Merged

UebelAndre closed this Nov 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rust Analyzer: Generate rust-project.json without explicit target list #907

Rust Analyzer: Generate rust-project.json without explicit target list #907

djmarcin commented Aug 23, 2021 •

edited

Loading

UebelAndre left a comment

UebelAndre Aug 23, 2021

djmarcin Aug 23, 2021

hlopko commented Aug 24, 2021

djmarcin commented Aug 30, 2021

UebelAndre commented Sep 8, 2021

hlopko left a comment

hlopko Sep 9, 2021

hlopko Sep 9, 2021

hlopko Sep 9, 2021

djmarcin Sep 13, 2021 •

edited

Loading

djmarcin left a comment

djmarcin Sep 13, 2021 •

edited

Loading

UebelAndre commented Sep 24, 2021

purkhusid commented Oct 4, 2021

hlopko commented Oct 4, 2021

purkhusid commented Oct 4, 2021

hlopko commented Oct 8, 2021

UebelAndre commented Oct 9, 2021

djmarcin commented Oct 26, 2021

UebelAndre commented Oct 26, 2021

UebelAndre Nov 5, 2021

UebelAndre commented Nov 17, 2021

djmarcin commented Nov 17, 2021


		return [OutputGroupInfo(rust_analyzer_sysroot_src = depset([sysroot_src_file]))]

		rust_analyzer_detect_sysroot = rule(

Rust Analyzer: Generate rust-project.json without explicit target list #907

Rust Analyzer: Generate rust-project.json without explicit target list #907

Conversation

djmarcin commented Aug 23, 2021 • edited Loading

UebelAndre left a comment

Choose a reason for hiding this comment

UebelAndre Aug 23, 2021

Choose a reason for hiding this comment

djmarcin Aug 23, 2021

Choose a reason for hiding this comment

hlopko commented Aug 24, 2021

djmarcin commented Aug 30, 2021

UebelAndre commented Sep 8, 2021

hlopko left a comment

Choose a reason for hiding this comment

hlopko Sep 9, 2021

Choose a reason for hiding this comment

hlopko Sep 9, 2021

Choose a reason for hiding this comment

hlopko Sep 9, 2021

Choose a reason for hiding this comment

djmarcin Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

djmarcin left a comment

Choose a reason for hiding this comment

djmarcin Sep 13, 2021 • edited Loading

Choose a reason for hiding this comment

UebelAndre commented Sep 24, 2021

purkhusid commented Oct 4, 2021

hlopko commented Oct 4, 2021

purkhusid commented Oct 4, 2021

hlopko commented Oct 8, 2021

UebelAndre commented Oct 9, 2021

djmarcin commented Oct 26, 2021

UebelAndre commented Oct 26, 2021

UebelAndre Nov 5, 2021

Choose a reason for hiding this comment

UebelAndre commented Nov 17, 2021

djmarcin commented Nov 17, 2021

djmarcin commented Aug 23, 2021 •

edited

Loading

djmarcin Sep 13, 2021 •

edited

Loading

djmarcin Sep 13, 2021 •

edited

Loading