-
Notifications
You must be signed in to change notification settings - Fork 13k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Diffs from UI tests are noisy when line numbers change #46643
Comments
We could filter out line numbers, but then moving something across the line 9/10 border will still produce noise because of the additional indent To make sure that tests where line numbers are important still can be tested we could have a flag that skips the filter step |
Alternatively we could replace line numbers by line offsets to the previous diagnostic's line number. |
Any line number can be replaced with a constant-width placeholder:
Yes, that's what I meant by "by default".
This mostly solves the noise problem, but I suspect the resulting numbers will be more misleading than actually useful. |
We could introduce a flag (or the opposite of it) // redact-line-numbers then during the normalization step, it will replace all However, the line 9/10 problem still happens due to alignment:
Note the column position of the arrow
|
I think the best solution would be the unstable compiler flag that does the replacement upfront (as suggested by @petrochenkov ), because of the large number of hacks needed to make the other solutions work.
|
Don't we want to keep line numbers for at least the header lines ( --> $DIR/issue-43253.rs:49:9), to prevent confusion if error messages are reordered? |
Anonymize some line numbers in UI test output New unstable flag `-Z ui-testing` is introduced. This flag changes diagnostic output of the compiler *in some way* making it more suitable for UI testing (this is intentionally vague). At the moment this flag anonymizes line numbers at line starts thus solving the largest issue with UI test diffs. If diffs continue to be too noisy, some other tweaks could be applied (e.g. anonymizing lines/columns in `--> $DIR/file.rs:line:column`), but this needs some time and experience (we shouldn't diverge too much from the actual output in general). If comment `// disable-ui-testing-normalization` is added to an UI test, then `-Z ui-testing` is not passed. Closes rust-lang#46643
Anonymize some line numbers in UI test output New unstable flag `-Z ui-testing` is introduced. This flag changes diagnostic output of the compiler *in some way* making it more suitable for UI testing (this is intentionally vague). At the moment this flag anonymizes line numbers at line starts thus solving the largest issue with UI test diffs. If diffs continue to be too noisy, some other tweaks could be applied (e.g. anonymizing lines/columns in `--> $DIR/file.rs:line:column`), but this needs some time and experience (we shouldn't diverge too much from the actual output in general). If comment `// disable-ui-testing-normalization` is added to an UI test, then `-Z ui-testing` is not passed. Closes #46643
Here's a typical example: #46640
One line added at the start of the test source (
// must-compile-successfully
) causes changes in all line numbers in the test's output.There are probably tests in which specific values of line numbers are important, but maybe they can be replaced with some placeholder by default?
The text was updated successfully, but these errors were encountered: