Do not deduplicate diagnostics in UI tests #67122

petrochenkov · 2019-12-07T16:10:50Z

Error reporting infrastructure deduplicates identical diagnostics with identical spans.

While it's preferable to do this in "release"/"user-facing" mode, it sometimes brings confusion and hides details that may be important during development.

Do we run some passes multiple times when we could do it once?
How many times we run them exactly? Can this number be large? Can the multiplied error construction be expensive? Can speculative checks be made cheaper if they don't report errors?

Relying on this mechanism to deduplicate some specific error never looks like a proper solution to me personally.

In this PR I attempt to disable this deduplication by applying -Z deduplicate-diagnostics=no to UI tests.

petrochenkov · 2019-12-07T16:12:21Z

@rust-lang/wg-diagnostics @rust-lang/compiler
What do you think about the general idea of disabling this deduplication, and a specific mechanism of doing it?

estebank · 2019-12-07T19:24:05Z

I'm only worried about adding more divergence between the testing stderr output and what users see. Would it be reasonable to make this a -Zdo-not-deduplicate flag that can be opted in by tests to make it clearer that's what's happening?

Centril · 2019-12-07T19:43:21Z

This seems like a great idea to me.

Would it be reasonable to make this a -Zdo-not-deduplicate flag that can be opted in by tests to make it clearer that's what's happening?

Imo the default should be to not de-duplicate since, at least in my view, the primary function of tests is to ensure correctness and not diagnostics quality. Where there are specific tests which look at diagnostics quality they can use a flag.

estebank · 2019-12-07T23:22:30Z

@Centril that's fair, having the opposite default in tests, and have the tests opt-in to the deduplication would be fine.

nikomatsakis · 2019-12-10T22:20:41Z

Hmm, I'm torn. I find it very useful to get an accurate picture of what the diagnostic output is for every test -- and I think it's been quite useful from time to time in catching unexpected regressions in diagnostic quality.

However, most of those regressions would be caught with or without this de-duplication. And, as @petrochenkov pointed out, there may be other sorts of regressions (running things twice etc) that we would catch this way. (Though perhaps also by monitoring perf.)

One question on the PR, does this mean we have to add corresponding //~ ERROR annotations?

In any case, I guess this is the key point:

Relying on this mechanism to deduplicate some specific error never looks like a proper solution to me personally.

I'm not sure if this is true, I can see an argument for it, but I can imagine there are some examples where it is very hard to suppress duplicates, particularly given the query infrastructure (i.e., the desire to compute separate things independently without unnecessary coordination).

I guess that if it is a bug for this mechanism to kick in (which doesn't quite seem to be what we're suggesting) then this definitely makes sense. I think what we're saying is more like "it's ok to rely on this mechanism, but it's still useful to detect cases where it kicks in and make them visible"?

Anway, I don't really have a strong opinion one way or the other about the change itself, so long as we're not calling all of these cases "bugs" (that part I am not yet convinced of).

bors · 2019-12-26T07:00:45Z

☔ The latest upstream changes (presumably #67268) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · 2019-12-28T09:22:46Z

Relevant Zulip discussion - https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/weekly.20meeting.202019-12-19.20.2354818/near/183854498.

The conclusion seems to be to proceed with this PR as is, with -Z ui-testing triggering the behavior change.

I was going to add a separate option like -Z deduplicate-diagnostics=yes/no, but perhaps it's just an extra unnecessary entity.

petrochenkov · 2019-12-29T06:56:32Z

On the other hand, tying this to -Z ui-testing will mean inability to enable the deduplication for individual tests without disabling the line anonymization as well.

petrochenkov · 2019-12-29T21:06:17Z

Blocked on #67709.

Introduce an option for disabling deduplication of diagnostics With the intent of using it in UI tests (rust-lang#67122). The option is boolean (`-Z deduplicate-diagnostics=yes/no`) and can be specified multiple times with later values overriding earlier values (`-Z deduplicate-diagnostics=no -Z deduplicate-diagnostics=yes` == `-Z deduplicate-diagnostics=yes`), so it can be set in a hierarchical way, e.g. UI testing infra may disable the deduplication by default with specific tests being able to enable it back.

petrochenkov · 2020-01-08T17:04:08Z

Updated.
r? @estebank

petrochenkov · 2020-01-08T17:10:22Z

Looks like the worst offenders are:

Lint level attribute checking (didn't investigate).
Derive macro resolution (this is a known inefficiency, derive macros are resolved multiple times).
Some checks for patterns in constants like structural-match or floating point patterns (didn't investigate).

Some cases are legitimately different errors, just reported for the same span, usually due to macros.

estebank · 2020-01-08T18:10:08Z

@petrochenkov the changes look good to me, but I'm a bit uneasy about silent regressions in the deduplication. Could you take a test for each of those three identified cases and make them deduplicate? (I think for one of them it is already the case.) That way the majority of the tests will not hide the duplication but we'll have a way to see if we inadvertently regress.

r=me with that change.

petrochenkov · 2020-01-08T20:30:20Z

@bors r=estebank

bors · 2020-01-08T20:30:22Z

📌 Commit d426771 has been approved by estebank

Do not deduplicate diagnostics in UI tests Error reporting infrastructure deduplicates identical diagnostics with identical spans. While it's preferable to do this in "release"/"user-facing" mode, it sometimes brings [confusion](rust-lang#50682 (comment)) and hides details that may be important during development. Do we run some passes multiple times when we could do it once? How many times we run them exactly? Can this number be large? Can the multiplied error construction be expensive? Can speculative checks be made cheaper if they don't report errors? *Relying* on this mechanism to deduplicate some specific error never looks like a proper solution to me personally. In this PR I attempt to disable this deduplication by applying `-Z deduplicate-diagnostics=no` to UI tests.

Centril · 2020-01-09T11:38:24Z

Failed in #68044 (comment), @bors r-

petrochenkov · 2020-01-09T18:50:02Z

@bors r=estebank

bors · 2020-01-09T18:50:03Z

📌 Commit b82cd9f has been approved by estebank

Do not deduplicate diagnostics in UI tests Error reporting infrastructure deduplicates identical diagnostics with identical spans. While it's preferable to do this in "release"/"user-facing" mode, it sometimes brings [confusion](rust-lang#50682 (comment)) and hides details that may be important during development. Do we run some passes multiple times when we could do it once? How many times we run them exactly? Can this number be large? Can the multiplied error construction be expensive? Can speculative checks be made cheaper if they don't report errors? *Relying* on this mechanism to deduplicate some specific error never looks like a proper solution to me personally. In this PR I attempt to disable this deduplication by applying `-Z deduplicate-diagnostics=no` to UI tests.

@ghost

Rollup of 10 pull requests Successful merges: - #66254 (Make Layout::new const) - #67122 (Do not deduplicate diagnostics in UI tests) - #67358 (Add HashSet::get_or_insert_owned) - #67725 (Simplify into_key_slice_mut) - #67935 (Relax the Sized bounds on Pin::map_unchecked(_mut)) - #67967 (Delay bug to prevent ICE in MIR borrowck) - #67975 (Export public scalar statics in wasm) - #68006 (Recognise riscv64 in compiletest) - #68040 (Cleanup) - #68054 (doc: add Null-unchecked version section to mut pointer as_mut method) Failed merges: - #67258 (Introduce `X..`, `..X`, and `..=X` range patterns) r? @ghost

petrochenkov added A-diagnostics Area: Messages for errors, warnings, and lints S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 7, 2019

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). labels Dec 28, 2019

petrochenkov mentioned this pull request Dec 29, 2019

Introduce an option for disabling deduplication of diagnostics #67709

Merged

petrochenkov added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Dec 29, 2019

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-blocked Status: Blocked on something else such as an RFC or other implementation work. labels Jan 5, 2020

petrochenkov changed the title ~~[experiment] Do not deduplicate diagnostics in -Z ui-testing mode~~ Do not deduplicate diagnostics in UI tests Jan 5, 2020

petrochenkov force-pushed the nodedup branch from d850f76 to d2d5fe9 Compare January 8, 2020 17:02

rust-highfive assigned estebank Jan 8, 2020

petrochenkov added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 8, 2020

petrochenkov force-pushed the nodedup branch from d2d5fe9 to d426771 Compare January 8, 2020 20:30

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 8, 2020

Centril mentioned this pull request Jan 9, 2020

Rollup of 6 pull requests #68044

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jan 9, 2020

petrochenkov added 3 commits January 9, 2020 21:23

compiletest: Do not deduplicate diagnostics in UI tests

41318e9

Update tests

642669c

Address review comments + Update NLL tests

b82cd9f

petrochenkov force-pushed the nodedup branch from d426771 to b82cd9f Compare January 9, 2020 18:49

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jan 9, 2020

JohnTitor mentioned this pull request Jan 9, 2020

Rollup of 10 pull requests #68067

Merged

bors merged commit b82cd9f into rust-lang:master Jan 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not deduplicate diagnostics in UI tests #67122

Do not deduplicate diagnostics in UI tests #67122

petrochenkov commented Dec 7, 2019 •

edited

Loading

petrochenkov commented Dec 7, 2019

estebank commented Dec 7, 2019

Centril commented Dec 7, 2019

estebank commented Dec 7, 2019

nikomatsakis commented Dec 10, 2019

bors commented Dec 26, 2019

petrochenkov commented Dec 28, 2019

petrochenkov commented Dec 29, 2019

petrochenkov commented Dec 29, 2019

petrochenkov commented Jan 8, 2020

petrochenkov commented Jan 8, 2020

estebank commented Jan 8, 2020

petrochenkov commented Jan 8, 2020

bors commented Jan 8, 2020

Centril commented Jan 9, 2020

petrochenkov commented Jan 9, 2020

bors commented Jan 9, 2020

Do not deduplicate diagnostics in UI tests #67122

Do not deduplicate diagnostics in UI tests #67122

Conversation

petrochenkov commented Dec 7, 2019 • edited Loading

petrochenkov commented Dec 7, 2019

estebank commented Dec 7, 2019

Centril commented Dec 7, 2019

estebank commented Dec 7, 2019

nikomatsakis commented Dec 10, 2019

bors commented Dec 26, 2019

petrochenkov commented Dec 28, 2019

petrochenkov commented Dec 29, 2019

petrochenkov commented Dec 29, 2019

petrochenkov commented Jan 8, 2020

petrochenkov commented Jan 8, 2020

estebank commented Jan 8, 2020

petrochenkov commented Jan 8, 2020

bors commented Jan 8, 2020

Centril commented Jan 9, 2020

petrochenkov commented Jan 9, 2020

bors commented Jan 9, 2020

petrochenkov commented Dec 7, 2019 •

edited

Loading