Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: report ERROR wpt test results #50429

Closed
wants to merge 2 commits into from

Conversation

legendecas
Copy link
Member

When a wpt test file is exited for uncaught error, its result should be
recorded in the wptreport.json and uploaded to wpt.fyi.

For instance, html/webappapis/timers/evil-spec-example.any.js is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to the tests. labels Oct 27, 2023
@legendecas legendecas force-pushed the wpt/report branch 2 times, most recently from 780117a to 111c759 Compare October 27, 2023 15:27
@legendecas
Copy link
Member Author

@panva It would be great if you could take a look at this when you get a chance since you originally created wpt.fyi report workflow. Thank you!

@panva panva self-requested a review October 27, 2023 17:38
@panva
Copy link
Member

panva commented Oct 27, 2023

evaluation in WPTRunner.runJsTests() gets reported as a test in the ERRORed file, see https://staging.wpt.fyi/results/resource-timing/delivery-type.tentative.any.html?run_id=5544793184665600&run_id=5129270332489728, i ran your PR against staging.wpt.fyi

I believe it should just be the harness that has an error and then no tests in it, like so https://wpt.fyi/results/streams/readable-byte-streams/construct-byob-request.any.html?run_id=5105883752693760&run_id=5146271209226240&run_id=5855453021470720&run_id=5148705381941248&run_id=6261279079006208

Same for timeout in our case.

@legendecas
Copy link
Member Author

Right. I've removed the synthetic sub-tests from the report to wpt.fyi. Thanks for the suggestion.

@panva
Copy link
Member

panva commented Oct 28, 2023

I'll re-run the upload tonight to check what it looks like now.

@panva
Copy link
Member

panva commented Oct 30, 2023

The evaluation in WPTRunner.runJsTests() looks alright now, it's gone.

But why is there a newly added for entry for all idlharness-shadowrealm.window.html which isn't present on main and wasn't present before the last fixup?

When a wpt test file is exited for uncaught error, its result should be
recorded in the `wptreport.json` and uploaded to wpt.fyi.

For instance, `html/webappapis/timers/evil-spec-example.any.js` is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.
@legendecas
Copy link
Member Author

Well, it seems a completion with error status was not recognized as a failed test and it was not reported to wpt.fyi. Fixed with e842497.

@legendecas legendecas added commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. request-ci Add this label to start a Jenkins CI on a PR. labels Oct 30, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Oct 30, 2023
@panva panva added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 30, 2023
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@legendecas legendecas added the commit-queue Add this label to land a pull request using GitHub Actions. label Nov 3, 2023
@nodejs-github-bot nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Nov 3, 2023
@nodejs-github-bot
Copy link
Collaborator

Commit Queue failed
- Loading data for nodejs/node/pull/50429
✔  Done loading data for nodejs/node/pull/50429
----------------------------------- PR info ------------------------------------
Title      test: report ERROR wpt test results (#50429)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     legendecas:wpt/report -> nodejs:main
Labels     test, author ready, needs-ci, commit-queue-rebase
Commits    2
 - test: report error wpt test results
 - test: recognize wpt completion error
Committers 1
 - legendecas 
PR-URL: https://github.com/nodejs/node/pull/50429
Reviewed-By: Filip Skokan 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/50429
Reviewed-By: Filip Skokan 
--------------------------------------------------------------------------------
   ℹ  This PR was created on Fri, 27 Oct 2023 15:23:01 GMT
   ✔  Approvals: 1
   ✔  - Filip Skokan (@panva): https://github.com/nodejs/node/pull/50429#pullrequestreview-1704170475
   ✘  This PR needs to wait 0 more hours to land (or 0 hours if there is one more approval)
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2023-11-01T06:12:59Z: https://ci.nodejs.org/job/node-test-pull-request/55378/
- Querying data for job/node-test-pull-request/55378/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu
https://github.com/nodejs/node/actions/runs/6746717439

@legendecas legendecas removed the commit-queue-failed An error occurred while landing this pull request using GitHub Actions. label Nov 3, 2023
@legendecas legendecas added the commit-queue Add this label to land a pull request using GitHub Actions. label Nov 3, 2023
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Nov 3, 2023
@nodejs-github-bot
Copy link
Collaborator

Landed in fd2368e...a00f0b1

nodejs-github-bot pushed a commit that referenced this pull request Nov 3, 2023
When a wpt test file is exited for uncaught error, its result should be
recorded in the `wptreport.json` and uploaded to wpt.fyi.

For instance, `html/webappapis/timers/evil-spec-example.any.js` is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.

PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
nodejs-github-bot pushed a commit that referenced this pull request Nov 3, 2023
PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
@legendecas legendecas deleted the wpt/report branch November 3, 2023 16:12
anonrig pushed a commit to anonrig/node that referenced this pull request Nov 9, 2023
When a wpt test file is exited for uncaught error, its result should be
recorded in the `wptreport.json` and uploaded to wpt.fyi.

For instance, `html/webappapis/timers/evil-spec-example.any.js` is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.

PR-URL: nodejs#50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
anonrig pushed a commit to anonrig/node that referenced this pull request Nov 9, 2023
PR-URL: nodejs#50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
targos pushed a commit that referenced this pull request Nov 11, 2023
When a wpt test file is exited for uncaught error, its result should be
recorded in the `wptreport.json` and uploaded to wpt.fyi.

For instance, `html/webappapis/timers/evil-spec-example.any.js` is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.

PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
targos pushed a commit that referenced this pull request Nov 11, 2023
PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
targos pushed a commit that referenced this pull request Nov 14, 2023
When a wpt test file is exited for uncaught error, its result should be
recorded in the `wptreport.json` and uploaded to wpt.fyi.

For instance, `html/webappapis/timers/evil-spec-example.any.js` is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.

PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
targos pushed a commit that referenced this pull request Nov 14, 2023
PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
UlisesGascon pushed a commit that referenced this pull request Dec 11, 2023
When a wpt test file is exited for uncaught error, its result should be
recorded in the `wptreport.json` and uploaded to wpt.fyi.

For instance, `html/webappapis/timers/evil-spec-example.any.js` is
exited for uncaught error in Node.js but it shows as "MISSING" at
https://wpt.fyi/results/html/webappapis/timers?label=master&label=experimental&product=chrome&product=node.js&aligned.

PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
UlisesGascon pushed a commit that referenced this pull request Dec 11, 2023
PR-URL: #50429
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
@UlisesGascon UlisesGascon mentioned this pull request Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. needs-ci PRs that need a full CI run. test Issues and PRs related to the tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants