Specify requirements for test suite and test metadata

[csarven]

This issue is an ACTION of https://github.com/solid/test-suite-panel/blob/main/meetings/2022-05-13.md#proposal-2-specify-requirements-for-test-suite-and-test-metadata due 2022-05-27. To document a mutual understanding (see #considerations) towards the Test Suite Panel Charter proposal.

Related issues:

• Specify requirements and criteria for publishing test summaries and reports
• Specify criteria for assessing tests and results

Considerations:

1. What information should be available about the design of the test suite and tests?
2. How can tests be written and run?
3. How can the expected behaviour for tests be verified?
4. How can tests be corrected?
5. How to find contributors to review tests?
6. How can implementers be notified about test results?
7. How to communicate test coverage?
8. How to attribute test authors and reviewers?
9. ...

[michielbdejong]

Good idea to write this down more precisely!

[csarven]

Status of this comment: Draft

1. Metadata about the test suites (e.g., license, date, written by, reviewed by), the software (e.g., repository, version, maintainers), setup (e.g., platform, configuration), provenance (e.g., activity, entity, agent), input about environment (e.g., combination of subject and setup). Meet the requirements of reporting (issue 5) and test review checklist (issue 7).
2. Documentation on:

• how to set up a test environment, how to build the test system (if necessary).
• how to write tests that are in general, short, self-contained, and maybe provide best practices and guidelines.
• how to run tests (provide a set of steps to test and obtain the result). Tests may be run in different ways depending on the specification requirement, e.g., command-line, containers, Web browser.

3. Do preliminary checks against multiple implementations to catch anomalies or to tease out issues with the tests themselves. Tests authors and specification authors should coordinate regarding the test design.
4. PR the updated test, mark what it replaces, mark it as to be reviewed, request reviews.
5. Notify specification authors and editors (and other group of contributors) about new tests and request reviews, e.g., tagging on GitHub.
6. Tagging the project maintainer (issue 5).
7. Provide information indicating to what extent the test suite completely or proportionally covers the specification it aims to support (issue 5).
8. Link to their WebID, if not available, GitHub account.