Submitting WPT results to wpt.fyi for comparison

[panva]

Is there interest in adding Node.js to the list of products that wpt.fyi supports displaying results and comparisons for?

If I understood the task right, it would require a daily CI job which checks out the WPT tests using a branch name that the wpt repo pushes to on a daily schedule, called epochs/daily, executing our WPT runner, collecting its results, transforming and submitting them to the wpt.fyi together with metadata about the used Node.js version.

This would be done in a matrix for all maintained release lines which wpt.fyi calls stable channels, plus one for the experimental channel which would be our main branch checkout. So the job would e.g. checkout node's v18.x branch HEAD~1, updated the checked out WPTs to epochs/daily, ran the WPTs, transformed the results and submitted them.

To submit the results we'd request an identifier/username to be created in the wpt.fyi API, an API key for that username would be given to us to be stored as a Github Actions Repository Secret.

We would still only be running the WPT suites we have an existing checkout for.

Example how Deno has their results submitted

[anonrig]

This is pretty interesting. I wonder what we are missing...

[panva]

Just the extract and transform addition to make test-wpt to get results in a format the wpt.fyi API expects.

I'm happy to take this effort on if there's interest.

[anonrig]

I'm pretty much interested in this @panva. Do you need any help?

[panva]

@anonrig

1. figure out the expected result format we have to emulate, since we're not using the official WPT runner we need to write this part ourselves. We'll have to update our runner to spit out something that can be parsed and transformed into the expected format.
2. the actual Github CI job
3. request credentials so that the Node.js project can be added as a runner that can submit to wpt.fyi's API.
4. start publishing daily

I'm currently at step 1. I found a suggested JSON schema for the report format. I plan to look at what Deno does in this regard.

It would be immensely helpful if we had a report at hand., I can reach out to the Deno folk for getting one that they submit. Generated one from Deno https://gist.github.com/panva/f00124fb416dd8572b5008c4ab6e9dcc

[panva]

Working in https://github.com/panva/node/commits/wptfyi and testing via https://github.com/nodejs/node-auto-test/actions/workflows/daily-wpt-fyi.yml

[panva]

I've merged the job. Starting tonight, latest tomorrow night, there will be nightly results uploaded to staging.wpt.fyi

I will request production credentials next week.

[anonrig]

Thank you @panva. Great job!

[panva]

Hi Filip,

The production service now has the same username/password for your project. We haven't deployed the code there just yet, but it should be OK for you to start submitting data.

Thanks,
Panos

@targos can you please reconfigure the WPT_FYI_ENDPOINT variable on the nodejs/node repository to https://wpt.fyi/api/results/upload?

We should leave the nodejs/node-auto-test configuration as-is, I'll be extending our report with fetch-related WPTs from undici and will use that to test.

[targos]

Variable updated.

[panva]

Thank you. This is now completed. I will followup with a PR to nodejs/undici and nodejs/node to include Fetch API related WPTs to the submitted report.

[panva]

🎉 Node.js is now selectable in the wpt.fyi production Dashboard as a "Product".

We are submitting nightly and current builds daily already.

In progress is

• adding lts/* ([v18.x backport] test: add WPTRunner support for variants and generating WPT reports #46767)
• adding lts/-1 ([v16.x backport] test: add WPTRunner support for variants and generating WPT reports #46768)
• adding fetch related WPTs (tools: add undici WPTs to daily WPT Report #46763)