Migrate compiletest to use ui_test-style //@ directives

[jieyouxu]

Preface

There's an on-going effort to rewrite parts of or the entirety of compiletest
(rust-lang/compiler-team#536). A step towards this involve migrating
compiletest tests to use the ui_test framework, which
involves changing compiletest directives in // <directive-name> style to ui_test
//@ <directive-name> style (rust-lang/compiler-team#512).

This PR aims to implement the directive-style change from // to //@ for the remaining
non-"ui" test suite tests.

Key Changes

1. All tests/ tests now use //@ directives.
2. Compiletest only accepts //@ and issues an error if an old-style directive is detected.
3. // ignore-tidy and // ignore-tidy-* are considered tidy directives and are ignored by
   compiletest header parsing.

Diff Generation

The diff is generated by:

• Collecting directives from tests/ via hijacking compiletest to emit successfully parsed
  directive lines.
• Using a migration tool
  (https://github.com/jieyouxu/compiletest-ui_test-header-migration/tree/master) to replace
  // directives in compiletest tests with //@.

Reproduction Steps

0. Delete the temporary file $RUSTC_REPO_PATH/build/<target_triple>/test/__directive_lines.txt,
   if the collection script was previously ran.

1. Use the https://github.com/jieyouxu/rust/tree/collect-test-directives collect-test-directives
   script, which outputs a temporary file recording headers occuring in each compiletest test.

   • You need to checkout this branch: git checkout collect-test-directives.

   • This needs to be rebased on latest master to ensure up-to-date test directives can be collected.

   • You need to run ./x test on each of the test/* subfolders once:

     ./x test tests/assembly/ --stage 1 --force-rerun
     ./x test tests/codegen/ --stage 1 --force-rerun
     ./x test tests/codegen-units/ --stage 1 --force-rerun
     ./x test tests/coverage/ --stage 1 --force-rerun
     ./x test tests/coverage-run-rustdoc/ --stage 1 --force-rerun
     ./x test tests/debuginfo/ --stage 1 --force-rerun
     ./x test tests/incremental/ --stage 1 --force-rerun
     ./x test tests/mir-opt/ --stage 1 --force-rerun
     ./x test tests/pretty/ --stage 1 --force-rerun
     ./x test tests/run-make/ --stage 1 --force-rerun
     ./x test tests/run-make-fulldeps/ --stage 1 --force-rerun
     ./x test tests/run-pass-valgrind/ --stage 1 --force-rerun
     ./x test tests/rustdoc/ --stage 1 --force-rerun
     TARGET=<target-triple> ./x test tests/rustdoc-gui/ --stage 1 --force-rerun
     ./x test tests/rustdoc-js/ --stage 1 --force-rerun
     ./x test tests/rustdoc-js-std/ --stage 1 --force-rerun
     ./x test tests/rustdoc-json/ --stage 1 --force-rerun
     ./x test tests/rustdoc-ui/ --stage 1 --force-rerun
     ./x test tests/ui/ --stage 1 --force-rerun
     ./x test tests/ui-fulldeps/ --stage 1 --force-rerun

2. Checkout the migrate-compiletest-directives branch.

3. Run the migration tool https://github.com/jieyouxu/compiletest-ui_test-header-migration.

4. Check that the migration at least does not cause test failures if you change compiletest to
   accept //@ directives only. This is also required if the test outputs somehow need to be
   blessed.

   • RUSTC_TEST_FAIL_FAST=1 ./x test tests/<secondary-directory> --stage 1 --bless

5. Confirm that there is no difference after running the migration tool when you are on the
   migrate-compiletest-directives branch.

Follow Up Work

• Adjust rustc-dev-guide docs for compiletest directives (this time for all the other suites and modes). Update compiletest directives to be in ui_test style //@ rustc-dev-guide#1895.

[rustbot]

Some changes occurred in GUI tests.

cc @GuillaumeGomez

[Zalathar]

FYI, the coverage tests will need to have their .coverage snapshots re-blessed (or edited) as well.

Also watch out, because the part of a coverage test that uses those snapshots doesn't run in PR CI jobs, and by default it doesn't run in local builds either.

(See https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Building.20the.20profiler.20runtime.20in.20more.20profiles.2Fjobs.3F for more context on why.)

So you will probably need to set [build] profiler = true in your cargo.toml, to avoid unpleasant surprises when the full CI suite fails.

[jieyouxu]

FYI, the coverage tests will need to have their .coverage snapshots re-blessed (or edited) as well.

Also watch out, because the part of a coverage test that uses those snapshots doesn't run in PR CI jobs, and by default it doesn't run in local builds either.

(See https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Building.20the.20profiler.20runtime.20in.20more.20profiles.2Fjobs.3F for more context on why.)

So you will probably need to set [build] profiler = true in your cargo.toml, to avoid unpleasant surprises when the full CI suite fails.

Thanks for the heads up. I'm also expecting other failures too that'll only get picked up in full build lol (just like in #120881).

Since the coverage reports seem to have a reasonably predictable format, I think I might be able to hack up some external script to convert the contained directives.

[Zalathar]

Since the coverage reports seem to have a reasonably predictable format, I think I might be able to hack up some external script to convert the contained directives.

Yeah, for the changes you're making this approach should work pretty well.

[bjorn3]

Is it still the case that this is not an exhaustive list? If not can you remove the respective part of this comment.

[jieyouxu]

It's as exhaustive as the currently used directives go. It is still possible that some combination e.g. ignore-platform-xxx has never been used else where in the test suite.

[bors]

📌 Commit cd25009 has been approved by oli-obk

It is now in the queue for this repository.

[bors]

⌛ Testing commit cd25009 with merge f62f490...

[bors]

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing f62f490 to master...

[rust-timer]

Finished benchmarking commit (f62f490): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.2%	[2.0%, 2.4%]	2
Improvements ✅ (primary)	-2.8%	[-3.3%, -2.3%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.8%	[-3.3%, -2.3%]	2

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 649.98s -> 649.375s (-0.09%)
Artifact size: 310.97 MiB -> 310.95 MiB (-0.01%)

[klensy]

Compiletest only accepts //@ and issues an error if an old-style directive is detected.

Don't see any error now when running COMPILETEST_FORCE_STAGE0=1 python x.py --stage=0 t tests/run-make/issue-85441 --force-rerun

compiletest directives in tests/run-make Makefiles currently not migrated, is it intentionally left for other pr?

[jieyouxu]

Compiletest only accepts //@ and issues an error if an old-style directive is detected.

Don't see any error now when running COMPILETEST_FORCE_STAGE0=1 python x.py --stage=0 t tests/run-make/issue-85441 --force-rerun

compiletest directives in tests/run-make Makefiles currently not migrated, is it intentionally left for other pr?

The compiletest directives in Makefiles that use # are not migrated, because... the plan is rip the Makefiles out in favor of plain rust files, i.e. #113026. That PR is still on-going.