[browser] test robot! #123753
[browser] test robot! #123753
Conversation
pavelsavara
added
NO-REVIEW
github-actions
bot
added
the
needs-area-label
|
Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara |
There was a problem hiding this comment.
Pull request overview
This PR introduces a small “browser test robot” workflow for Browser/WASM + CoreCLR library tests, including helper scripts, documentation, and checked-in sample results for a few suites, plus a minor docs tweak around disabling tests.
Changes:
- Add a Browser/WASM CoreCLR test workflow: setup instructions, execution guide, and a plan document under
browser-tests/. - Introduce scripts to run browser tests locally, download Mono baselines from Helix, and compare CoreCLR vs Mono test sets, along with checked-in example result artifacts (logs, XML, summaries, and failure docs).
- Adjust CI documentation for disabling tests to reference skip attributes, including Mono-specific skips (with one malformed example that should be corrected).
Reviewed changes
Copilot reviewed 19 out of 34 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| docs/workflow/ci/disabling-tests.md | Documents how to disable runtime tests using xUnit attributes; one example for skipping on Mono is currently malformed (OnMono] instead of [SkipOnMono]). |
| browser-tests/before-testing.md | Provides one-time setup instructions for running Browser/WASM + CoreCLR tests (HTTPS dev cert, initial browser build, environment variables). |
| browser-tests/test-suite.md | Defines the end-to-end process for running individual browser test suites, collecting outputs, documenting failures, and maintaining per-suite summaries and comparisons. |
| browser-tests/run-browser-test.sh | Bash helper to run a single Browser/WASM CoreCLR test suite via ./dotnet.sh build /t:Test, capturing console output and copying xharness testResults.xml and wasm-console.log into browser-tests/results/<TestProject>/. |
| browser-tests/download-mono-baseline.sh | Bash script to pull the Mono Helix baseline for a given test project by querying Mono-chrome-workitems.json with jq and curl, storing mono-console.log and mono-testResults.xml under browser-tests/results/<TestProject>/. |
| browser-tests/compare-test-results.sh | Bash script that extracts test names from Mono vs CoreCLR testResults.xml, computes differences with comm, prints a human-readable summary, and writes test-comparison.txt into browser-tests/results/<TestProject>/. |
| browser-tests/plan.md | High-level plan and progress tracker for Browser/WASM + CoreCLR library tests, including platform characteristics, goals, and a status table for completed suites. |
| browser-tests/failures/System.Net.WebSockets.Tests/GetImplMethodDesc.Assert.md | Records a non-fatal CoreCLR interpreter assert (GetImplMethodDesc) observed after WebSockets tests complete, including stack trace and categorization. |
| browser-tests/results/.gitignore | Overrides the repository-wide ignore rules to ensure .log and .xml result files under browser-tests/results/ are kept in version control. |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/wasm-console_20260129_160809.log | Captured Browser/WASM + CoreCLR xharness console output for System.Runtime.InteropServices.JavaScript.Tests as a reference run. |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/test-comparison.txt | Machine-generated comparison of Mono vs CoreCLR test names for System.Runtime.InteropServices.JavaScript.Tests (showing extra/missing tests). |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/mono-console.log | Helix Mono baseline console log for System.Runtime.InteropServices.JavaScript.Tests, used by the comparison tooling. |
| browser-tests/results/System.Runtime.InteropServices.JavaScript.Tests/Summary.md | Hand-authored summary of the JavaScript interop tests’ CoreCLR vs Mono results, including benign differences and new tests list. |
| browser-tests/results/System.Net.WebSockets.Tests/test-comparison.txt | Machine-generated comparison of Mono vs CoreCLR for System.Net.WebSockets.Tests confirming identical test sets. |
| browser-tests/results/System.Net.WebSockets.Tests/mono-testResults.xml | Helix Mono baseline testResults.xml for the WebSockets tests, feeding the comparison and summary docs. |
| browser-tests/results/System.Net.WebSockets.Tests/mono-console.log | Helix Mono baseline console log for WebSockets tests. |
| browser-tests/results/System.Net.WebSockets.Tests/Summary.md | Summary of WebSockets CoreCLR vs Mono results and a link to the recorded interpreter assert. |
| browser-tests/results/System.Net.Http.Functional.Tests/wasm-console_20260129_162929.log | Captured console for a CoreCLR Browser/WASM run of System.Net.Http.Functional.Tests that discovered zero tests (Debug configuration / assembly-level skip). |
| browser-tests/results/System.Net.Http.Functional.Tests/testResults_20260129_162929.xml | Corresponding zero-test CoreCLR testResults.xml for System.Net.Http.Functional.Tests (Debug run). |
| browser-tests/results/System.Net.Http.Functional.Tests/mono-console.log | Helix Mono baseline console log for System.Net.Http.Functional.Tests. |
| browser-tests/results/System.Net.Http.Functional.Tests/mono-testResults.xml | Helix Mono baseline testResults.xml for System.Net.Http.Functional.Tests used in comparisons. |
| browser-tests/results/System.Net.Http.Functional.Tests/console_20260129_162929.log | Local build+test console for the zero-test CoreCLR Debug run of System.Net.Http.Functional.Tests. |
| browser-tests/results/System.Net.Http.Functional.Tests/testResults_20260129_163140.xml | CoreCLR testResults.xml for a subsequent run of System.Net.Http.Functional.Tests (non-zero test count; used by comparison and summary). |
| browser-tests/results/System.Net.Http.Functional.Tests/wasm-console_20260129_163140.log | Browser/WASM + CoreCLR console output for the non-zero-test run of System.Net.Http.Functional.Tests. |
| browser-tests/results/System.Net.Http.Functional.Tests/test-comparison.txt | Comparison of Mono vs CoreCLR test names for System.Net.Http.Functional.Tests (differences are only in server port values). |
| browser-tests/results/System.Net.Http.Functional.Tests/Summary.md | Summary of Http functional tests on Browser/WASM + CoreCLR, explaining configuration requirements and interpreting comparison results. |
| browser-tests/results/System.Net.WebSockets.Tests/mono-testResults.xml | (Same as above row for this path) Baseline WebSockets test result data used to power comparisons and summaries. |
| grep -oP 'name="[^"]*"' "$xml_file" | \ | ||
| grep -v 'assembly name=' | \ | ||
| grep -v 'collection.*name=' | \ | ||
| sed 's/name="//;s/"$//' | \ | ||
| sort -u | ||
| } | ||
|
|
There was a problem hiding this comment.
The filtering of assembly and collection names happens after grep -oP 'name="[^"]*"', so those lines no longer contain the assembly or collection text and are not actually excluded; this causes assembly and collection names to be included as "tests" and skews the Mono/CoreCLR counts (e.g., the MetricOuterLoop1 comparison reports 1 test even though total="0" in the XML). Restrict the extraction to <test ...> elements before pulling the name attribute (for example by grepping for <test first or by adjusting the regex to only match name="..." on <test> lines) so only real test cases are counted.
| grep -oP 'name="[^"]*"' "$xml_file" | \ | |
| grep -v 'assembly name=' | \ | |
| grep -v 'collection.*name=' | \ | |
| sed 's/name="//;s/"$//' | \ | |
| sort -u | |
| } | |
| grep -oP '<test\b[^>]*\Kname="[^"]*"' "$xml_file" | \ | |
| sed 's/name="//;s/"$//' | \ | |
| sort -u | |
| } |
| @@ -0,0 +1,16 @@ | |||
| ## Rebuild After Changes | |||
There was a problem hiding this comment.
The comment says "Rebuild just libraries" but the command uses -subset clr+libs+host, which also rebuilds the host; either update the text to reflect that host is included or change the subset to clr+libs if the intention is truly to rebuild only libraries. Aligning the wording and the command will help avoid confusion for people following these instructions.
browser-tests/one.md
Outdated
| Read the `browser-tests/plan.md` | ||
| Run the steps in `browser-tests/before-testing.md`. | ||
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | ||
|
|
||
|
|
||
| Read the `browser-tests/plan.md` | ||
| Run the steps in `browser-tests/before-testing.md` except building the runtime, that already done. | ||
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests` | ||
|
|
||
| Read the `browser-tests/plan.md` | ||
| The before testing steps are already done. | ||
| Now we want to run `browser-tests/test-suite.md` process for `System.Collections.Immutable.Tests` No newline at end of file |
There was a problem hiding this comment.
This file repeats similar instructions three times with slightly different preconditions and target test suites, which makes it unclear whether these are sequential steps, alternative flows, or just notes; consolidating into a single, clearly structured set of steps (or explicitly labeling them as separate scenarios) would make it easier for others to understand and follow.
| Read the `browser-tests/plan.md` | |
| Run the steps in `browser-tests/before-testing.md`. | |
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | |
| Read the `browser-tests/plan.md` | |
| Run the steps in `browser-tests/before-testing.md` except building the runtime, that already done. | |
| Then we want to run `browser-tests/test-suite.md` process for `System.Runtime.Tests` | |
| Read the `browser-tests/plan.md` | |
| The before testing steps are already done. | |
| Now we want to run `browser-tests/test-suite.md` process for `System.Collections.Immutable.Tests` | |
| This document describes three alternative scenarios for running browser tests. | |
| ## Scenario 1: First-time run of System.Runtime.Tests | |
| 1. Read `browser-tests/plan.md`. | |
| 2. Run all steps in `browser-tests/before-testing.md`. | |
| 3. Run the `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | |
| ## Scenario 2: Runtime already built, rerun System.Runtime.Tests | |
| Use this scenario if the runtime has already been built, but the other "before testing" steps still need to run. | |
| 1. Read `browser-tests/plan.md`. | |
| 2. Run the steps in `browser-tests/before-testing.md` except for building the runtime. | |
| 3. Run the `browser-tests/test-suite.md` process for `System.Runtime.Tests`. | |
| ## Scenario 3: Before-testing steps already completed, run System.Collections.Immutable.Tests | |
| Use this scenario if all steps in `browser-tests/before-testing.md` have already been completed. | |
| 1. (Optional) Re-read `browser-tests/plan.md` if you need a refresher. | |
| 2. Run the `browser-tests/test-suite.md` process for `System.Collections.Immutable.Tests`. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 59 out of 1213 changed files in this pull request and generated 8 comments.
Comments suppressed due to low confidence (3)
browser-tests/results/Microsoft.Bcl.Numerics.Tests/Summary.md:1
- The header claims 882 total tests with 882 passed, while the comparison section states 874 Mono/CoreCLR tests and the note says the counts "match Mono baseline exactly". These numbers are inconsistent; either the total should be 874 or the comparison values need adjusting so that the summary and baseline comparison agree.
# Microsoft.Bcl.Numerics.Tests - Browser WASM Test Results
browser-tests/results/Invariant.Tests/Summary.md:1
- Here the total test count is listed as 397, but the comparison section and note say there are 380 Mono/CoreCLR tests and that the counts "match Mono baseline exactly". It would be good to reconcile these numbers (e.g., by using 380 throughout or explaining any intentional discrepancy) so the summary is self-consistent.
# Invariant.Tests - Browser WASM Test Results
browser-tests/results/Microsoft.Bcl.AsyncInterfaces.Tests/Summary.md:1
- The header and XML results show 55 total tests for this run, but the comparison section reports 59 tests for both Mono and CoreCLR, and the note claims the counts "match Mono baseline exactly." Please align the comparison counts with the actual 55-test run (or clarify if a different counting convention is intended) so the summary isn't contradictory.
# Microsoft.Bcl.AsyncInterfaces.Tests - Browser WASM Test Results
| # Find the most recent CoreCLR test results | ||
| CORECLR_RESULTS=$(ls -t "${RESULTS_DIR}"/testResults_*.xml 2>/dev/null | head -1) |
There was a problem hiding this comment.
With set -e enabled at the top of the script, this ls will exit with a non-zero status (and terminate the script immediately) when no testResults_*.xml files exist, so the user never sees the friendly error message below. Consider using a more robust pattern (e.g., shopt -s nullglob then iterating the glob, or using find/printf), or explicitly guarding this command so that a missing file does not trigger set -e.
| # Find the most recent CoreCLR test results | |
| CORECLR_RESULTS=$(ls -t "${RESULTS_DIR}"/testResults_*.xml 2>/dev/null | head -1) | |
| # Find the most recent CoreCLR test results | |
| set +e | |
| CORECLR_RESULTS=$(ls -t "${RESULTS_DIR}"/testResults_*.xml 2>/dev/null | head -1) | |
| set -e |
| Mono tests: 16 | ||
| CoreCLR tests: 16 |
There was a problem hiding this comment.
The Mono/CoreCLR test counts shown here (16) do not match the actual total=\"14\" counts in the corresponding mono-testResults.xml and testResults_*.xml files, which both report 14 tests. Please update these values (and regenerate this comparison file if needed) so that the documented counts are consistent with the recorded test results.
| Mono tests: 16 | |
| CoreCLR tests: 16 | |
| Mono tests: 14 | |
| CoreCLR tests: 14 |
| Mono tests: 59 | ||
| CoreCLR tests: 59 |
There was a problem hiding this comment.
This comparison reports 59 tests for Mono/CoreCLR, but the mono-testResults.xml and CoreCLR testResults_20260130_133146.xml assemblies both declare total=\"55\". To avoid confusion, the counts here should be aligned with the XML totals (or the comparison should be regenerated from the current XML).
| Mono tests: 59 | |
| CoreCLR tests: 59 | |
| Mono tests: 55 | |
| CoreCLR tests: 55 |
| Mono tests: 1 | ||
| CoreCLR tests: 1 |
There was a problem hiding this comment.
Both the Mono and CoreCLR *testResults*.xml files for MetricOuterLoop1.Tests have total=\"0\" and no <test> elements, while this comparison reports 1 test on each side. Please update these counts to reflect that no tests actually ran in these Browser/WASM runs.
| Mono tests: 1 | |
| CoreCLR tests: 1 | |
| Mono tests: 0 | |
| CoreCLR tests: 0 |
| Mono tests: 4 | ||
| CoreCLR tests: 4 |
There was a problem hiding this comment.
The Mono and CoreCLR *testResults*.xml files for MetricOuterLoop.Tests each report total=\"2\" tests (one passed, one skipped), whereas this comparison file states 4 tests. The comparison should be regenerated or edited so that the test counts match the XML data (2, not 4).
| Mono tests: 4 | |
| CoreCLR tests: 4 | |
| Mono tests: 2 | |
| CoreCLR tests: 2 |
| Mono tests: 3 | ||
| CoreCLR tests: 3 |
There was a problem hiding this comment.
The IcuAppLocal Mono and CoreCLR result files each show total=\"1\" and a single skipped test, but this comparison reports 3 tests on each side. Please fix these counts to match the actual number of tests recorded in the XML (1).
| Mono tests: 3 | |
| CoreCLR tests: 3 | |
| Mono tests: 1 | |
| CoreCLR tests: 1 |
| ## Latest Run | ||
| - **Date:** 2026-01-30 | ||
| - **CoreCLR:** Tests run: 14, Passed: 8, Failed: 0, Skipped: 6 | ||
| - **Mono Baseline:** Tests run: 14, Passed: 8, Failed: 0, Skipped: 6 |
There was a problem hiding this comment.
This summary correctly reflects 14 tests (8 passed, 6 skipped), but the corresponding test-comparison.txt file currently reports 16 tests for both Mono and CoreCLR. To avoid confusion, either regenerate the comparison file from the current XML or update it so that the reported test counts match this summary.
| - **Mono Baseline:** Tests run: 14, Passed: 8, Failed: 0, Skipped: 6 | |
| - **Mono Baseline:** Tests run: 14 (per current XML; `test-comparison.txt` still lists 16 and should be regenerated), Passed: 8, Failed: 0, Skipped: 6 |
| - **Mono tests**: ~267 | ||
| - **CoreCLR tests**: 270 |
There was a problem hiding this comment.
The summary says 266 tests ran (239 passed, 27 skipped), but the comparison section claims 270 CoreCLR tests and approximately 267 Mono tests. These numbers should be made consistent—either by correcting the comparison counts or clarifying what the 270/267 figures represent relative to the 266-test run described above.
| - **Mono tests**: ~267 | |
| - **CoreCLR tests**: 270 | |
| - **Mono baseline tests (non-Browser)**: 267 total | |
| - **CoreCLR baseline tests (non-Browser)**: 270 total | |
| - This Browser/WASM run executed 266 of these tests (239 passed, 27 skipped) due to traits/environment filtering |
1 participant
No description provided.