-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Request: mark test runner stable in Node 20.0.0 #46642
Comments
+1 |
I haven't been tracking closely but I get the impression people open feature requests for it quite a bit. How would moving it out of experimental interact with those? |
Feature requests would have to follow semver. Of course we aren't obligated to implement all of the feature requests too - there are a few open feature requests that I personally don't think we should implement. |
cc @nodejs/test_runner I think this should be fine. We could just create experimental flags or warnings for any parts that are remaining experimental or that we think might get breaking changes. So really that’s my only question: are there any parts of the test runner, besides code coverage, that might get breaking changes? Reporters? How about we isolate those under experimental flags so that it’s clear to users when they’re deviating from the stable path, and call the rest stable. |
Is there a way we could measure "ready" beyond just "i feel"? It'd be great to try to collect feedback about who has migrated to node's test runner, and who hasn't and why not. |
I think the test api and runner are stable. The reporters and codecov are too new to be stable, however they have a very little (if at all) API surface. I’d recommend we keep the codecov and reporters as doc-experimental until 20.0.0 reach LTS but otherwise drop all flags and warnings. From a marketing perspective, announcing v20 with test runner is a great announcement. |
I'd like to bring up #44023 again. I haven't cared about this topic much since opening that issue due to being busy with other things, but I consider that problem largely unsolved. The PR that closed that issue (#44241) is nice for sure but doesn't address command-line usage, and so IMHO doesn't really improve the TypeScript DX that much. Stabilizing could prevent such improvements from happening since they'd possibly be breaking changes. Please reconsider adding support for a string argument to To recap: The test runner has hardcoded globs for the test files. It doesn't support running test files written in TypeScript or any other language, or otherwise filtering the test files, without (1) listing all of them explicitly, or (2) writing a custom entry point. (1) is near impossible to maintain, and (2) requires significant amounts of boilerplate code and/or third-party packages, which defeats the point of a native test runner. |
@meyfa could you open a new issue with the things that are missing? Note that marking it stable doesn't mean we cannot add things, but rather that we won't break users. |
there is still an open issue on the subject #46292. |
Marking tsc-agenda for visibility. |
Would adding glob support be a breaking change in some way? Other than “before it errored and now it doesn’t.” As far as I’m concerned we should identify any parts that might still get breaking changes and wall those off as experimental, and mark the remainder as stable. I also think we should add warnings or experimental flags for the parts that might still change, because with the bulk of the test runner being stable it shouldn’t fall on users to pore through the docs to ensure that they’re only using the stable parts of the runner. I would imagine that there would likely be many users who only want to use the stable parts, waiting for the experimental parts to become stable, because they don’t want to deal with updating their testing code often as the experimental parts change. |
It would be a breaking change if we use $ rm -rf repro && mkdir repro && cd repro
$ mkdir "**"
$ echo 'console.log("pass")' > "**/*.js"
$ echo 'throw new Error' > "**/should-fail.js"
$ node --test **/*.js
TAP version 13
# Subtest: …/repro/**/*.js
# pass
ok 1 - …/repro/**/*.js
---
duration_ms: 112.994764
...
1..1
# tests 1
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 115.977816
$ cd .. && rm -r repro If we change the meaning of |
It’s a bit ridiculous to consider a file named |
One thing to note is that changing the files to search would likely involve a semver minor change to add a new CLI flag, and not a breaking change to |
I think the best path forward is to list all the concerns / missing features as issues and track them accordingly, so we can stabilize it in a short timeframe. |
Somewhere, the issue of our increasingly bloated CLI was raised. The test mode and the watch mode alone have added about ten CLI options or so to the CLI. In a related discussion (which I also can't find right now), someone suggested a I don't have a strong opinion on this at this point, but I think it should at least be noted if either mode is considered for stable. |
@tniessen this thread is not related to Including the experimental code coverage flag, the test runner has added six CLI flags. For comparison, there are 7 flags for just the
That would not work because all of the other test runner flags can be used without It is definitely possible for Node to scope CLI flags, and that would be a breaking change, but I think that would be a reasonable breaking change to ship separately (and as I mentioned above, not really apply to the test runner currently).
The CLI will likely keep getting bigger, but I don't think it's fair to try to attribute all of that to the test runner. Last I checked, the Deno executable was bigger than the Node executable. If the CLI getting larger is a pressing matter, I think we either need to stop shipping new features in core or find a way to ship core outside of the CLI (a topic for another issue though). |
Thanks for the detailed comment @cjihrig. I neither have much insight nor a strong opinion on this, but I am glad it has been considered :) |
Would loader also be marked stable in v20? If yes, then this would mean we can run TypeScript tests directly with Node, right? This woud be so great! |
What is the problem this feature will solve?
Having a stable test runner in core. People have been asking for it, and I feel the majority of the test runner is ready.
What is the feature you are proposing to solve the problem?
I would like to stabilize the test runner for the 20.0.0 release. This currently would not include code coverage, which is behind a separate flag.
What alternatives have you considered?
Not stabilizing the test runner.
The text was updated successfully, but these errors were encountered: