Close the front door for clippy but open the back #7533

yaahc · 2019-10-22T14:58:24Z

No description provided.

rust-highfive · 2019-10-22T14:58:35Z

(rust_highfive has picked a reviewer for you, use r? to override)

yaahc · 2019-10-22T15:03:30Z

Feeling pretty good about this PR atm, the biggest problem right now is that I broke rustfix isolation, so now its fixing the non relevant crates. I think this is because I didn't scope the fixing to primary units like how I adjusted things in an earlier change, and rustfix probably had some separate hacky way of telling it to only fix primary crates, where as my last changes made it so we just invoke rustc rather than cargo rustc on non primary crates.

I'm also kinda confused on whether or not I should be using the term primary_crates or workspace_crates. We historically use primary but in the recent meeting we used workspace, so I defaulted with the one that aligned with the name we wanted to give the env variable to control the workspace rustc path.

To continue with the pattern we've established here I think we need a WORKSPACE_RUSTFLAGS to allow clippy to still pass args to the driver, originally I expected that the place for this would be in clippy as something like CLIPPYFLAGS or something but @Manishearth informs me that RUSTFLAGS is a cargo thing, so it seems like the sensible thing to do here is be consistent so I'll look into implementing that next.

ehuss · 2019-10-22T17:53:44Z

I'm also kinda confused on whether or not I should be using the term primary_crates or workspace_crates

Ah, sorry about being vague there. To be clear, "primary packages" are those selected on the command line. For example, cargo build -p foo -p bar, the primary packages are foo and bar, and nothing else.

I think that opens the question on how clippy should work. And maybe how cargo fix works as well, now that I'm looking at it closer (it ignores the target), although it probably isn't too important (nobody has ever complained). Fix should continue to only run against the primary package. As for clippy, I see a few options:

Run only against primary packages like fix does. This is how clippy-preview was working.
Run against all workspace packages. The reason I was suggesting this is because that is how normal lints work (when you build in a workspace, you get warnings for any workspace member). This is also how the stable cargo clippy works (although it runs on all dependencies, and then the lints are capped on non-workspace members).

So I guess the question boils down to, do you want to see clippy lints for dependencies in a workspace? I don't have a good answer. The primary package seems more useful (think of a workspace like rustc where there are dozens of dependencies). On the other hand, that is inconsistent with how normal lints work.

I have to leave for a while, but I'll try to do some more review here tonight.

yaahc · 2019-10-22T19:31:16Z

I definitely prefer it to run against all workspace crates, but not necessarily path dependencies, which is something I think it currently does. I've had cases where I checked out a git repo for an unpublished crate and depended on it with a path dep and it surfaced clippy lints which was not what I wanted.

I'm not actually sure how cargo differentiates between these two cases. Ive only seen things talking about primary crates in the source, so I'll have to figure out how it deliniates crates that are in the workspace specifically.

I'll look into filtering by workspace members for clippy and move forward with the assumption that this is how we want to filter application of the alternate rustc unless I get comments to the contrary.

I'll make sure to keep fix on primaries and try to add some test cases that configure various forms of projects with path deps, workspace members, primary crates specified on the cli when called, etc.

Manishearth · 2019-10-22T20:56:09Z

Ideally I think the answer would be "all workspace projects, unless you've specified a single one via -p", but "all workspace projects" is fine too.

ehuss · 2019-10-23T01:40:15Z

all workspace crates, but not necessarily path dependencies

Path dependencies are automatically added as workspace members. This can be overridden by setting the workspace.exclude key to remove the package as a member, which would cap its lints.

how cargo differentiates between these two cases

Workspace members are computed in Workspace::find_members. The primary packages are computed here. The Units to compile is computed in two stages. The first generates the root units in generate_targets, which is where the "primary units" comes from (I would say "root" is synonymous with "primary"). The second phase is computing all dependencies from the roots in unit_dependencies.

Ideally I think the answer would be "all workspace projects, unless you've specified a single one via -p", but "all workspace projects" is fine too.

Cargo doesn't really have this kind of distinction. If you don't specify -p, there is an implied -p that is injected very early. I would also think the inconsistency might be surprising.

I think this is essentially rust-lang/rust-clippy#3025, right? I don't have any good answers on how to address this. Maybe we could set an env var that the clippy wrapper could detect (and the clippy wrapper could maybe have a --no-deps flag to restrict where it runs), but we recently rejected (or postponed) such a thing in #7205 (comment).

(Sorry, running out of energy for today, will try to look closer at the code tomorrow.)

Manishearth · 2019-10-23T01:47:30Z

I don't think it's a big deal if we can't solve clippy#3025 right now; I'd rather get this to work for workspaces and figure that out later

ehuss

Seems pretty close. Just to be clear, I think the precedence of which rustc is used would be:

BuildConfig.primary_unit_rustc
RUSTC_WORKSPACE_WRAPPER
RUSTC_WRAPPER
RUSTC
rustc in path

Where primary_unit_rustc is only used by cargo fix to set cargo itself as the wrapper.

Also, we'll probably want to remove clippy from CI (azure-test-all.yml), as I think it will no longer be needed.

src/cargo/core/compiler/build_config.rs

src/cargo/core/compiler/build_context/mod.rs

src/cargo/core/compiler/compilation.rs

src/cargo/core/compiler/context/compilation_files.rs

src/cargo/ops/fix.rs

src/cargo/util/config/mod.rs

bors · 2019-10-28T20:21:42Z

☔ The latest upstream changes (presumably #7550) made this pull request unmergeable. Please resolve the merge conflicts.

yaahc · 2019-11-21T01:32:33Z

src/cargo/util/rustc.rs

+            .filter(|wrapper| !wrapper.get_program().is_empty())
+            .map(|wrapper| {
+                let mut cmd = wrapper.clone();
+                // TODO: FIX HACK


This is where having the primary_unit_rustc exist as a wrapper becomes a problem. cargo fix doesn't really want to use the rustc_wrapper either, it's only supposed to run fix for primary units so it makes sense to invoke it via the primary wrapper, but when you do that it turns it into a wrapper by adding the path (rustc) as an arg, but for fix we already added the path to the wrapper so you end up with cargo clippy-driver rustc ... at the end. This check prevents that extra incorrect insert from happening but we need to find a better way to do this.

src/cargo/util/rustc.rs

ehuss · 2019-11-22T01:45:02Z

IIUC, this is moving in a slightly different direction where it is only running against primary units, not all workspace packages, correct? If that is true, calling it a "workspace wrapper" may not be entirely correct. I'm also not sure if that is what @Manishearth wanted (the message at #7533 (comment) seemed to imply all workspace crates is preferred). I'd also be a little concerned this is encroaching on the concerns from #7205 (comment), where "primary" isn't well established.

primary packages are strictly a subset of the workspace ones

Not really. -p can be specified for any package.

Whichever approach is taken, it would be good to have some tests for the new behavior.

Manishearth · 2019-11-22T01:46:30Z

Yeah, I think we wanted: - all workspace packages, if run from workspace root - all workspace packages being compiled, if run from a specific crate

…

On Thu, Nov 21, 2019, 5:45 PM Eric Huss ***@***.***> wrote: IIUC, this is moving in a slightly different direction where it is only running against primary units, not all workspace packages, correct? If that is true, calling it a "workspace wrapper" may not be entirely correct. I'm also not sure if that is what @Manishearth <https://github.com/Manishearth> wanted (the message at #7533 (comment) <#7533 (comment)> seemed to imply all workspace crates is preferred). I'd also be a little concerned this is encroaching on the concerns from #7205 (comment) <#7205 (comment)>, where "primary" isn't well established. primary packages are strictly a subset of the workspace ones Not really. -p can be specified for any package. Whichever approach is taken, it would be good to have some tests for the new behavior. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7533?email_source=notifications&email_token=AAMK6SFUNEPRYATHES4IAI3QU42R7A5CNFSM4JDRBOY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE4H3TQ#issuecomment-557350350>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMK6SDEHYJBLX742EMKMLTQU42R7ANCNFSM4JDRBOYQ> .

yaahc · 2019-11-22T19:22:45Z

Okay, yea I wasn't sure if we decided that primary crates were close enough to workspace crates or if we wanted to intentionally aim at workspace crates. In that case I shall figure out an is_workspace_unit function to replace the is_primary and rename all instances of primary that I've added to workspace.

bors · 2019-11-25T17:02:47Z

☔ The latest upstream changes (presumably #7628) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2019-12-16T16:12:15Z

☔ The latest upstream changes (presumably #7710) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2020-01-08T18:26:50Z

☔ The latest upstream changes (presumably #7776) made this pull request unmergeable. Please resolve the merge conflicts.

rustc-echo-wrapper

tests/testsuite/check.rs

src/cargo/core/compiler/mod.rs

rustc-echo-wrapper

tests/testsuite/check.rs

src/cargo/ops/fix.rs

ehuss

Thanks! I think this is getting pretty close. Some notes:

There should be some kind of test in testsuite/cache_messages.rs that verifies when the workspace wrapper is set that it rebuilds compared to when it is not set (essentially the same as the old clippy test). The wrapper will just need to emit some extra output.
Windows build is broken.
The notes at the bottom of Close the front door for clippy but open the back #7533 (comment) still need to be addressed.

Also, have you tested with sccache to see if it works with clippy?

src/cargo/ops/fix.rs

src/cargo/util/process_builder.rs

src/cargo/core/compiler/compilation.rs

src/cargo/core/compiler/context/compilation_files.rs

src/cargo/util/rustc.rs

tests/testsuite/fix.rs

src/cargo/ops/fix.rs

tests/testsuite/cache_messages.rs

bors · 2020-03-12T17:16:36Z

☔ The latest upstream changes (presumably #7838) made this pull request unmergeable. Please resolve the merge conflicts.

ehuss · 2020-03-14T20:50:11Z

OK! I think this looks good overall. Thanks so much for sticking with it @yaahc!

Do you want to work on the clippy side to enable this? I think there are two things to do:

With -Zunstable-options set, use RUSTC_WORKSPACE_WRAPPER here (if running on nightly channel).
Add a --fix flag to cargo-clippy that will change it to use "fix" here instead of "check". At least, that's how I think it should work, but of course converse with the clippy team what they think.

Unfortunately I don't see a way to check if clippy is on the nightly channel in the clippy codebase, but maybe it can access is_nightly_build directly from the rustc_session crate? I'm not too familiar with that.

There's no rush. The rust PR queue is somewhat backed up, so it may take a couple weeks for this to land on nightly.

@bors r+

bors · 2020-03-14T20:50:12Z

📌 Commit d9a77ce has been approved by ehuss

bors · 2020-03-14T20:50:20Z

⌛ Testing commit d9a77ce with merge 7302186...

bors · 2020-03-14T21:07:36Z

☀️ Test successful - checks-azure
Approved by: ehuss
Pushing 7302186 to master...

Update cargo Update cargo 21 commits in bda50510d1daf6e9c53ad6ccf603da6e0fa8103f..7019b3ed3d539db7429d10a343b69be8c426b576 2020-03-02 18:05:34 +0000 to 2020-03-17 21:02:00 +0000 - Run through clippy (rust-lang/cargo#8015) - Fix config profiles using "dev" in `cargo test`. (rust-lang/cargo#8012) - Run CI on all PRs. (rust-lang/cargo#8011) - Add unit-graph JSON output. (rust-lang/cargo#7977) - Split workspace/validate() into multiple functions (rust-lang/cargo#8008) - Use Option::as_deref (rust-lang/cargo#8005) - De-duplicate edges (rust-lang/cargo#7993) - Revert "Disable preserving mtimes on archives" (rust-lang/cargo#7935) - Close the front door for clippy but open the back (rust-lang/cargo#7533) - Fix CHANGELOG.md typos (rust-lang/cargo#7999) - Update changelog note about crate-versions flag. (rust-lang/cargo#7998) - Bump to 0.45.0, update changelog (rust-lang/cargo#7997) - Bump libgit2 dependencies (rust-lang/cargo#7996) - Avoid buffering large amounts of rustc output. (rust-lang/cargo#7838) - Add "Updating" status for git submodules. (rust-lang/cargo#7989) - WorkspaceResolve: Use descriptive lifetime label. (rust-lang/cargo#7990) - Support old html anchors in manifest chapter. (rust-lang/cargo#7983) - Don't create hardlink for library test and integrations tests, fixing rust-lang/cargo#7960 (rust-lang/cargo#7965) - Partially revert change to filter debug_assertions. (rust-lang/cargo#7970) - Try to better handle restricted crate names. (rust-lang/cargo#7959) - Fix bug with new feature resolver and required-features. (rust-lang/cargo#7962)

Remove clippy tests. Clippy was removed in #7533, but a stale file was left behind.

Stargateur · 2020-04-15T02:34:23Z

would have be nice to report this change in the changelog. matching the "Added cargo fix --clippy which will apply machine-applicable fixes from Clippy. #7069" in the change log.

ehuss · 2020-04-15T02:42:03Z

@Stargateur The changelog is updated 6 weeks before release. It will probably be updated next week for the release in June. Also, this isn't the complete solution. rust-lang/rust-clippy#5363 will need to be merged first, and then it will need to go through the stabilization process, so there isn't much to say in the changelog other than clippy-preview was removed.

Stargateur · 2020-04-15T02:44:31Z

@ehuss It would have remove the pain to understand why stable ask for nightly and then nightly say the option doesn't exist. And maybe that before merge this have the rust-lang/rust-clippy#5363 ready to merge too would have been better. Anyway I never used this option so /me fly away.

(Also, the actual (nightly) build doesn't have the option so I don't understand your "It will probably be updated next week for the release in June." there is a miss somewhere in your organisation)

rust-highfive assigned alexcrichton Oct 22, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Oct 22, 2019

ehuss assigned ehuss and unassigned alexcrichton Oct 22, 2019

ehuss reviewed Oct 23, 2019

View reviewed changes

ehuss mentioned this pull request Nov 1, 2019

RUSTC_WRAPPER not used in cargo clippy? #7556

Closed

yaahc force-pushed the clippy-banishment branch from 97b2a4f to 175238d Compare November 20, 2019 23:50

yaahc commented Nov 21, 2019

View reviewed changes

yaahc marked this pull request as ready for review November 21, 2019 01:33

yaahc commented Nov 21, 2019

View reviewed changes

src/cargo/util/rustc.rs Outdated Show resolved Hide resolved

ehuss mentioned this pull request Dec 3, 2019

cache-messages will cache incorrect messages when switching between cargo check and cargo clippy #7635

Closed

yaahc force-pushed the clippy-banishment branch from 397d6b5 to 7caefed Compare December 15, 2019 22:10

yaahc force-pushed the clippy-banishment branch from 7caefed to 4d999a6 Compare December 16, 2019 22:46

yaahc force-pushed the clippy-banishment branch from 4d999a6 to 4fc11a4 Compare January 13, 2020 21:37

yaahc commented Jan 13, 2020

View reviewed changes

rustc-echo-wrapper Outdated Show resolved Hide resolved

tests/testsuite/check.rs Outdated Show resolved Hide resolved

tests/testsuite/check.rs Outdated Show resolved Hide resolved

ehuss reviewed Jan 15, 2020

View reviewed changes

src/cargo/core/compiler/mod.rs Outdated Show resolved Hide resolved

rustc-echo-wrapper Outdated Show resolved Hide resolved

tests/testsuite/check.rs Outdated Show resolved Hide resolved

tests/testsuite/check.rs Outdated Show resolved Hide resolved

Manishearth mentioned this pull request Feb 21, 2020

CLIPPY_FLAGS to pass options to clippy/ enable disable lints via env vars rust-lang/rust-clippy#5214

Closed

yaahc force-pushed the clippy-banishment branch from 4ace5ef to 6a0f252 Compare March 1, 2020 20:34

yaahc commented Mar 1, 2020

View reviewed changes

src/cargo/ops/fix.rs Outdated Show resolved Hide resolved

ehuss reviewed Mar 5, 2020

View reviewed changes

yaahc commented Mar 5, 2020

View reviewed changes

tests/testsuite/cache_messages.rs Show resolved Hide resolved

Close the front door for clippy but open the back

b0351e4

yaahc force-pushed the clippy-banishment branch from c53f61a to b0351e4 Compare March 13, 2020 02:28

yaahc added 3 commits March 13, 2020 14:59

switch from bash script to rust program

76797b5

rustfmt

d02f476

fix unix cfg on include

d9a77ce

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 14, 2020

bors merged commit 7302186 into rust-lang:master Mar 14, 2020

bors mentioned this pull request Mar 14, 2020

Add unit-graph JSON output. #7977

Merged

ehuss mentioned this pull request Mar 18, 2020

Update cargo rust-lang/rust#69907

Merged

matthiaskrgr mentioned this pull request Mar 23, 2020

cargo fix cmd no longer works rust-lang/rust-clippy#5358

Closed

ehuss mentioned this pull request Mar 29, 2020

Remove clippy tests. #8053

Merged

bors added a commit that referenced this pull request Mar 30, 2020

Auto merge of #8053 - ehuss:remove-clippy, r=alexcrichton

0c0231e

Remove clippy tests. Clippy was removed in #7533, but a stale file was left behind.

ehuss mentioned this pull request Apr 21, 2020

RUSTC_WORKSPACE_WRAPPER tracking issue #8143

Closed

4 tasks

DianaNites mentioned this pull request Jan 26, 2021

Cargo clippy with cargo-sysroot warns on sysroot DianaNites/cargo-sysroot#6

Closed

ehuss added this to the 1.44.0 milestone Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Close the front door for clippy but open the back #7533

Close the front door for clippy but open the back #7533

yaahc commented Oct 22, 2019

rust-highfive commented Oct 22, 2019

yaahc commented Oct 22, 2019

ehuss commented Oct 22, 2019

yaahc commented Oct 22, 2019

Manishearth commented Oct 22, 2019

ehuss commented Oct 23, 2019

Manishearth commented Oct 23, 2019

ehuss left a comment

bors commented Oct 28, 2019

yaahc Nov 21, 2019

ehuss commented Nov 22, 2019

Manishearth commented Nov 22, 2019 via email

yaahc commented Nov 22, 2019

bors commented Nov 25, 2019

bors commented Dec 16, 2019

bors commented Jan 8, 2020

ehuss left a comment

bors commented Mar 12, 2020

ehuss commented Mar 14, 2020

bors commented Mar 14, 2020

bors commented Mar 14, 2020

bors commented Mar 14, 2020

Stargateur commented Apr 15, 2020

ehuss commented Apr 15, 2020

Stargateur commented Apr 15, 2020 •

edited

Loading

Close the front door for clippy but open the back #7533

Close the front door for clippy but open the back #7533

Conversation

yaahc commented Oct 22, 2019

rust-highfive commented Oct 22, 2019

yaahc commented Oct 22, 2019

ehuss commented Oct 22, 2019

yaahc commented Oct 22, 2019

Manishearth commented Oct 22, 2019

ehuss commented Oct 23, 2019

Manishearth commented Oct 23, 2019

ehuss left a comment

Choose a reason for hiding this comment

bors commented Oct 28, 2019

yaahc Nov 21, 2019

Choose a reason for hiding this comment

ehuss commented Nov 22, 2019

Manishearth commented Nov 22, 2019 via email

yaahc commented Nov 22, 2019

bors commented Nov 25, 2019

bors commented Dec 16, 2019

bors commented Jan 8, 2020

ehuss left a comment

Choose a reason for hiding this comment

bors commented Mar 12, 2020

ehuss commented Mar 14, 2020

bors commented Mar 14, 2020

bors commented Mar 14, 2020

bors commented Mar 14, 2020

Stargateur commented Apr 15, 2020

ehuss commented Apr 15, 2020

Stargateur commented Apr 15, 2020 • edited Loading

Stargateur commented Apr 15, 2020 •

edited

Loading