Audit run-fail ui tests to see if they need *exact* run stderr/stdout

[jieyouxu]

compiletest triage: I'm not convinced switching error-pattern run-fail ui tests to use only the current form of check-run-results is an improvement over error-pattern unconditionally. If anything, it would be a regression to flaky tests in many cases that are sensitive to exact run stdout/stderr (line numbers, backtraces that can depend on platform-specific unwind mechanism, concrete addresses, version numbers, etc.).

I would be in favor of auditing some of these run-fail tests and sparingly add check-run-results if the exact run stderr/stdout is critical for the intention of the test, but not in a blanket fashion. As such, I'm going to close this issue in favor of a more concrete audit issue.

Originally posted by @jieyouxu in #65865

[jieyouxu]

See also Audit {check-run-results, error-pattern, check-stdout, normalize-*, dont-check-compiler-*, run-rustfix, rustfix-only-machine-applicable, forbid-output} family of directives.

[jieyouxu]

E-medium because you need some experience to not accidentally make a test flaky and to know if a test needs to check for exact run stderr/stdout.

[Pyr0de]

Hey,
I'd like to give this issue a shot

By audit do mean I'll need to change the tests or just report on the tests that can use check-run-results? And if I do need to change them, then do I try to normalize output to get a consistent outputs?

[jieyouxu]

By audit, I meant that you'll need to double-check the tests if they really need exact run stderr, or if they actually just care about a specific error pattern. It's less about the actual snapshot, it's more about the test intention.

[Pyr0de]

Here's an example from what I've understood,
A test that tests panic would need the exact stderr but a test which would panic but the intent of the the test is not to test panic would not need exact stderr (not the best example but I hope it's clear enough)

[jieyouxu]

Here's an example from what I've understood, A test that tests panic would need the exact stderr but a test which would panic but the intent of the the test is not to test panic would not need exact stderr (not the best example but I hope it's clear enough)

Something like that yeah. The nuisance here is exactly determining which tests need the exact run stderr and which do not.

Even tests exercising panics need to be careful, exact backtraces can be very sensitive to platform differents or environment influences.

[Pyr0de]

There seem to be some tests which have both error-pattern and check-run-results, is there a reason for this?
Example 1 and 2

[jieyouxu]

Yes, actually. check-run-results will snapshot and compare the exact stderr. However, the exact stderr is easy to accidentally bless away. error-pattern or regex-error-pattern is there to additionally ensure the key things we're looking for are still there, i.e. harder to accidentally bless away. This can be useful if you consider that (1) the exact stderr should be snapshotted and (2) there are specific key bits that you want to ensure its presence.

If anything, usually only having check-run-results can be suspicious.

[Pyr0de]

I'm sorry but I don't think I will be able to solve this issue properly
Thanks for the all the help @jieyouxu

[jieyouxu]

No worries, it's not easy by any means