Introduce moz-webgpu-cts process-reports

[ErichDonGubler]

As the developer of Firefox implementation of WebGPU, it's sometimes difficult to quickly adjust test expectations after I make changes to Firefox. CTS comprises a large set of tests and files that I work with, and it can be time-consuming and even error-prone to find the right places in files to edit by hand when I receive WPT reports from CI.

I would like tool assistance in adjusting expectations in the correct locations, given a set of WPT reports in JSON format (i.e., wptreport.json) files that are provided by CTS runs in CI (./mach wpt --log-wptreport=…). I recognize that there are different heuristics that I may want to use for adjusting expected outcomes, so I'll offer users their choice of adjustment heuristic. The data I'll use is:

1. WPT metadata files underneath testing/web-platform/mozilla/meta/webgpu. Per test and subtest, there will be a set of expected outcomes per platform.
2. One or more WPT execution reports. For tests relevant to WebGPU, there will be tests within the webgpu test path that have actual outcomes on a specific platform noted. These can be combined to form a set of actual outcomes in a structure otherwise very similar to (1).

Overall flow of moz-webgpu-cts process-reports

The flow of the tool should be:

1. Gather a bunch of WPT reports using a tool like ./mach wpt-fetch-logs try:2639473c8ba1d33e13a3108db7660cc8cbed91f5 or (eventually) Add moz-webgpu-cts dl-reports subcommand #18.
2. Invoke moz-webgpu-cts process-reports --preset=… <wpt-report-json-paths…>
   1. Note that we are specifying a --preset=… here. This is a required argument to specify a heuristic for making changes based on reports. Presets are intended to be very coarse grained configurations that match typical workflows for convenience. We do not provide other options for finer-grained configuration at this time, but we plan to do so, and track progress on such at Tracking issue: moz-webgpu-cts update-expected options #26.
   2. Also accept 0 or more --glob <glob> options, which correspond to wax globs.
   3. Validate that the base name of each file path passed directly or yielded by the glob ends with wptreport.json. If any paths fail validation, emit a warning on stderr.
3. Gather data sets (1) and (2), as noted above. Data set (2) will be derived from file paths to JSON blobs containing WPT reports that are enumerated in the previous step.
   1. If there are any parsing or validation errors with either data set, then bail on the entire invocation with a set of error(s) that seem appropriate.
4. Compare expected test and subtest outcomes in metadata with reported outcomes by platform and build profile. Use the provided --preset=… to determine how to react in each comparison case.
   1. For this MVP, it is acceptable if all metadata files are re-emitted, even when not substantially changed; such a case would be as if somebody invoked the fmt subcommand (see Implement moz-webgpu-cts fmt for normalizing expectation formatting #8).
   2. EDIT: already implemented.

Technical notes

Fortunately, we have a lot of good data to work with. We already implement parsing of WPT metadata as of #5, so the remaining work is:

• Parse WPT execution reports from JSON
• Perform comparisons between expected and actual outcomes
• Adjust moz-webgpu-cts fmt to restructure metadata as described in "Overall flow of moz-webgpu-cts process-reports", requirement 4.ii. EDIT: Done with Normalize outcome order for tests and subtests #39 and Impose total ordering on sections in WebGPU CTS files #43.