Arbitrary annotations in compiletest

[pietroalbini]

Proposal

This MCP proposes to add a new compiletest declaration, annotations, that allows the person writing the test to add arbitrary annotations to the test. While those annotations won't impact the test execution, compiletest will emit a JSON file at the end of the test run containing all the annotations it found, and the checksums of all the files related to those tests.

For example, if you add this comment to the bastion of the turbofish test:

// annotations: foo

...compiletest will generate a JSON file like this at the end of the test run:

{
  "parser/bastion-of-the-turbofish.rs": {
    "annotations": ["foo"],
    "sha256": {
      "parser/bastion-of-the-turbofish.rs": "840ae8552ba944ae468df113f1ee4123b260fbb32a43f71af907950586848333"
    }
  }
}

In Ferrocene we need annotations because we need to establish a "traceability matrix" between some requirements and their related tests, to document that all of our requirements are properly tested and which files are part of the test. Compiletest annotations would generate all the necessary information we need on the test suite side of things, allowing our traceability matrix tool to parse the JSON output.

Note that even if this MCP is accepted we likely won't upstream the annotations themselves, as they'd reference internal documents and would be impossible to maintain outside of Ferrocene.

Still, the generic annotations mechanism could be useful in the Rust project as well, and we'd love to upstream it if this MCP is accepted. Some uses of annotations I could see are annotating which tests would need to be un-ignored when a target approaches Tier 1, or annotating tests with the issues they fix.

Mentors or Reviewers

@Mark-Simulacrum

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

cc @rust-lang/compiler @rust-lang/compiler-contributors

[apiraino]

MCP candidate for closing next week (reason: lack of seconding). See MCP process.

[jyn514]

@rustbot second

[apiraino]

@rustbot label -final-comment-period +major-change-accepted