Tracking Issue for assorted compiletest maintenance

[jieyouxu]

This is a tracking issue for a series of compiletest cleanups. This tracking issue is on-going and will be edited as suitable to reflect the next steps. Tasks should be broken up into small actionable items.

Motivation

Currently compiletest is a bit of a mess causing it to be really hard to maintain. Let's try to do some housekeeping in compiletest to make it easier to maintain.

Phase 1: compiletest/src/runtest.rs cleanups

• Step 1: Break up compiletest/src/runtest.rs into smaller helper modules. (Break up compiletest runtest.rs into smaller helper modules #130566)
  • Originally Splitting up compiletest::runtest #89475
• Step 2: Investigate and rework how valgrind test suites are handled, namely what happens if valgrind is not available.
  • Deleted the run-pass-valgrind test suite and valgrind support in Remove valgrind test suite and support from compiletest, bootstrap and opt-dist #131351.
  • Purge run-pass-valgrind mentions rustc-dev-guide#2091
• Step 3: Relocate functions on TestCx that does not need to be on TestCx (especially ones that don't depend on TestCx itself) to suitable locations.
• Step 4: Reorganize methods on TestCx:
  • Step 4.1: Privatize methods only used by a specific test suite/mode to their specific helper modules.
  • Step 4.2: Reorder/regroup methods on the core TestCx in runtest.rs to make it easier to navigate.
• Step 5: Improve documentation around runtest.rs:
  • Step 5.1: Make sure tool docs are registered for compiletest
    • Tool docs not registered for compiletest #130564
    • Register tool docs for compiletest #130567
  • Step 5.2: Document util and helper methods on TestCx.
  • Step 5.3: Document individual test suites/modes.
  • Step 5.4: Document top-level TestCx and types/concepts in runtest.rs.
  • Step 5.5: Update rustc-dev-guide docs about the individual test modes/suites and about test running.
  • Step 5.6: Add an example in rustc-dev-guide about how to add a new test suite/mode.
• Step 6: Review implementation of each test suite/mode.

Phase 2: Rework compiletest error handling and error reporting

• Step 1: Investigate how compiletest currently handles errors and reports them.
• Step 2: Come up with a design to make compiletest error reporting more cohesive and more helpful for users.
• TODO

Phase 3: Rework directive handling

• Step 1: Survey existing directive handling related bugs.
  • Step 1.1: Creating a tracking issue: Tracking Issue for compiletest directive handling bugs and papercuts #131425.
  • Step 1.2: Write a document to describe current limitations/problems with how directives are handled.
• Step 2: Redesign how directives are parsed and handled.
  • Step 2.1: Write a document to describe the considerations for directive handling.
  • Step 2.2: Draft a prototype toy impl for the reworked handling would look like.
  • Step 2.3: Draft a MCP to propose implementing the rework.
  • @tgross35 has some experimentation in Rewrite the compiletest directive parser #128070, this should get an MCP to receive more feedback from other compiler team members (TODO(@jieyouxu): write-up an MCP)
  • https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Test.20header.20commands.20could.20just.20be.20toml.3F proposes we can use structured TOML directives in doc comments. non-starter as it makes writing tests quite annoying.
• Step 3: Implement more robust directive handling.
  • Step 3.1: Implement the more robust design but not merge yet, as we need to...
  • Step 3.2: ... find out which tests contain invalid directives w.r.t. new directive handling and fix them.
  • Step 3.3: Investigate and improve testing for directive handling.
  • Step 3.4: Try to land the improved directive handling.
• Step 4: Improve directive documentation in source and in rustc-dev-guide:
  • Step 4.1: Unify terminology: stick with "directive", not also "header" or "comment" etc.
    • [Testing 2/2] Revise directives docs rustc-dev-guide#2089
  • Step 4.2: Document individual directives: syntax, applicable to which test suites, behavior
    • [Testing 2/2] Revise directives docs rustc-dev-guide#2089
    • This can be improved once directive handling is more robust
• TODO

There are more phases intended, but they are to be planned.

Discussions

Rubber-ducking thread: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/.28Rubberducking.29.20compiletest.20test.20discovery.20.2F.20directives