Great convergence effort/suggestion: Making QUnit proxy to node.js and deno testing APIs in these environments

[izelnakri]

What are you trying to do?

Hi @Krinkle , I've been experimenting with the new default node.js test runner recently and managed to come up universal QUnit API that can proxy to node.js or deno on demand. The proxy still needs more work but you can have a look here:

https://github.com/izelnakri/qunitx/blob/528c771f80e89c1c912513c586da0acaa686206b/test/inputs/file-test.js

This test suite proxies to node.js New Test Runner API: https://nodejs.org/api/test.html

I had few discoveries related to this, I'll still need to do more work but I've discovered the most important initial discoveries. I think we can make QUnit truly universal JS Test runner/standard by only a little surface level API changes. At least I think we should steer this project forward in the newer majors releases with this vision.

Both deno and node.js has an optional options argument to define concurrency, timeout, sanitizeOps etc arguments:

Node.js: https://nodejs.org/api/test.html#testname-options-fn
Deno: https://deno.com/manual@v1.25.2/testing/sanitizers#resource-sanitizer

I think being able to do them optionally as a 2nd argument to module and test is what we should do as well. Thats the only major change/deprecation I see for the current core testing API: https://nodejs.org/api/test.html#testname-options-fn .

We can add/remove other things later if we want to but thats the only change we have to do on our API surface side to eventually become a universal proxy API. module second hooks argument probably needs to be removed, it is redundant anyways considering hooks argument gets provided in the callback function, in this proposed new version test() API can accept three arguments to be fully customizable from outside.

Eventually I foresee node.js and deno will work together to standardize/converge their testing APIs, some effort has been done it seems on this front however deno's bdd & test modules/API design still seems custom: https://deno.com/manual@v1.25.2/testing/behavior_driven_development#nested-test-grouping

We can move both projects to come to a standard/testing API, or encourage the use of a unified tool(QUnit) that can proxy to their custom API and suggest changes to these projects.

Have a look at these references, and we can talk & discuss later if you wish. I've some more ideas I'm planning to implement for QUnitX but having a unified JS testing tool that works in nodejs, browser, deno with as little internal dependencies as possible is my current goal/ambition for QUnit & QUnitX and recently I discovered this will be possible.

[Krinkle]

@izelnakri Hey. Thanks for reaching out.

Let me echo some of this back to make sure I understand the broader idea and the specific proposal. The idea is to provide multiple implementations of the QUnit API (one for nodejs, one for browser, one for deno). For Deno and Node.js, these would not use the current test runner, but rather proxy to the new test runners built-in to Deno and Node.js, register each module and their tests there, and have that layer responsible for selecting, filtering, and running tests, and also let that layer control the output/results (i.e. serializing and formatting values for assertion failures, and TAP reporter).

The overaching purpose would be that applications and libraries that consider themselves isomorphic, can write their tests once and run them in all three kinds of environments.

Correct me if I got any of that wrong or otherwise differently from what you're thinking. So far, that sounds awesome to me. For test suites, things would be a lot simpler if test suites use ESM for discovering the source code imports, as that way they'll naturally work in both the browser and in Node.js/Deno.

Today most test runners require the HTML entrypoint and the CLI entrypoint to both sepecify the list of source files and test suits (though Karma and QUnit CLI both support glob to make this easier), and then there is Airtap which gets around this by standardising on CJS/ESM and runnning browserify for the browser target (though I dislike that as it adds overhead and complexity to both source and test code which make it harder to debug and less like how it runs normally in the browser).

If we add an options argument to QUnit.module() and QUnit.test(), the part that I'm not yet understanding, is what this would represent in the API. If it is a transparent pass-through to specific options that exist only in Deno, or only in Node.js, would that take away from the universality? E.g. if you set concurrency in your test, what would this do in the browser today? But.. maybe this is okay if the options are only management tweaks and e.g. not the kind of option that would be functionally observable and change behaviour such that it makes a test pass or fail depending on it.

Thinking more broadly about the strategy, would you mind drawing a comparison between a dream outcome and say what we have today with alternate approaches that run QUnit tests in Node + Browser. What benefits are you most excited about? E.g. something we can make possible and/or much easier.

[izelnakri]

Glad to hear you find this approach worthy for evaluation @Krinkle ! :)

If we add an options argument to QUnit.module() and QUnit.test(), the part that I'm not yet understanding, is what this would represent in the API. If it is a transparent pass-through to specific options that exist only in Deno, or only in Node.js, would that take away from the universality? E.g. if you set concurrency in your test, what would this do in the browser today? But.. maybe this is okay if the options are only management tweaks and e.g. not the kind of option that would be functionally observable and change behaviour such that it makes a test pass or fail depending on it.

options second argument module and test functions would indeed represent all the configuration that can be done to a specific test or module. Ideally they should use the defaults so they can be overriden by cli arguments: qunit test.ts --concurrency=0. This is needed because in the ideal world, the default would be that every test case is concurrent and parallel by default. Elixir programming language achieves this by running test suite by a random seed value and all the relevant test cases in that order, seed=0 means running everything in serial order like we do now by default. In node.js it functions takes its concurrency configuration from the above module unless specifically provided a concurrency: true option.

I imagine the options type interface difference between the node and deno test runners will converge, as both projects work together more. They do get inspired & evolve based on one another so I don't mind the mismatch for now, both configurations should be available in the same options argument.

Thinking more broadly about the strategy, would you mind drawing a comparison between a dream outcome and say what we have today with alternate approaches that run QUnit tests in Node + Browser. What benefits are you most excited about? E.g. something we can make possible and/or much easier.

I'm going to build QUnitX based on this strategy going forward and make it the test runner for my @memoria ORM project, it already runs in node.js and browser environments but soon I'll make some changes to make this shimming/proxying possible, also to make it work/tested on deno.

I'm mostly excited about having one flexible testing API for all environments & make tests completely concurrent/independent by default in JS. This will certainly allow us to deprecate bunch of internal packages in qunit as well eventually.

Testing is currently too hard as you mentioned especially in the frontend and even the big companies don't do enough testing or high quality testing. I see this as a major problem in the software industry now, if we can make it easier then adoption will certainly increase, or at least increase in areas/fields where it really matters. Even new hip alternatives like vitest considers browser mode a second class citizen and lacks many features QUnit already has for years, so this approach should help QUnit adoption as well. We don't need to add any more API than those options arguments to module and test, maybe deprecate more public API eventually.

I'll keep you posted about the @memoria/adapters test suite with the changes once I implement the shimming/proxying!