test: do not require flags when executing a file

[BridgeAR]

Instead of throwing an error in case a flag is missing, just start
a child_process that includes all flags. This improves the situation
for all developers in case they want to just plainly run a test.

Co-authored-by: Rich Trott rtrott@gmail.com

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

[nodejs-github-bot]

@BridgeAR build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/3008/pipeline

[sam-github]

I often run tests in a little harness that reruns them if the test code changes, or the node executable rebuilds, and run them directly with ./out/Release/node (not tools/test.py, its waaaayyyy slow), so this would be quite helpful to me, thanks.

[refack]

I'm ambivalent about this change. On the one hand it's convenient. On the other it's "magic" and definatly not K.I.S.S...

(not tools/test.py, its waaaayyyy slow)

Maybe put those lines in tools/test.js and then you'll be able to run:

node -r tools/test foo/test-barbaz.js

Make it a bit more picky and you can add it to your NODE_OPTIONS...

[BridgeAR]

@sam-github thanks for the review, I updated it similar to your suggestion. PTAL.

CI https://ci.nodejs.org/job/node-test-pull-request/21847/

@refack

it's "magic" and definatly not K.I.S.S...

Everything around our flags is currently magical. This just makes it a lot more convenient to run the tests and it is especially significantly friendlier to newcomers.

I also do not see that it makes the code more complex than it currently is, so is there any actual downside?

@nodejs/testing PTAL

[tniessen]

I often run gdb --args ./node_g test/parallel/test-something. Generally, this change sounds like a good idea, but it might make debugging confusing... I think.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/21874/

[sam-github]

Our javascript unit tests files are intended to be runnable standalone, because depency of tests on external runners introduces complexity and difficulty in debugging.

Usually they are runnable standalone, except sometimes they are only runnable with cooperation from a giant, undocumented, and difficult to understand glob of python (I say that as someone who has a half dozen times tried to figure out what some of its effectively undocumented command line arguments mean by reading the test.py source, unsucessfully). test.py seems to be the real black magic here.

I like this pure-node approach so much I'd be in favour of the dependency of our javascript tests on the python runner being stripped out, and this way of running the tests becoming the default.

As a developer friendly optimization, its great that the re-spawn of a child can be evaded when running with debuggers (gdb or inspector) by directly specifying the CLI options. This PR preserves that quality. 👍

[tniessen]

As a developer friendly optimization, its great that the re-spawn of a child can be evaded when running with debuggers (gdb or inspector) by directly specifying the CLI options.

My two cents: When debugging native code, I often use GDB with one of our tests and if I forget to specify flags, the test suite tells me. With this patch, when I forget to specify flags, I end up debugging the wrong process, and I don't think I'd notice that immediately (apart from "my breakpoints don't work"). Just a small concern, apart from that, it sounds like a great idea!

[BridgeAR]

@tniessen hm... I guess this is not a frequent use case for most users but it's definitely a very valid point. I guess we could have a blacklist for some environment variables or arguments that are used with GDB and the inspector. If anyone has a better idea, please let me know!

[BridgeAR]

I removed author ready until we have a solution for GDB.

[refack]

(I say that as someone who has a half dozen times tried to figure out what some of its effectively undocumented command line arguments mean by reading the test.py source, unsucessfully). test.py seems to be the real black magic here.

Challenge accepted 🔧
(I'd consider it more of a "black hole" then "black magic")

when I forget to specify flags, I end up debugging the wrong process

That's what I consider "magic". When the code tries to make non-trivial decisions for you...

IMO the flag parsing could go into the stdlib, similar to --exp█████████nals we could add a --run-test cli arg that triggers //FLAGS parsing in-proc.

[BridgeAR]

@refack I don't think that parsing our test flags justifies adding another flag and it also contradicts the actual intention here: to reduce the overhead. All new developers would have to learn about the CLI flag and would always have to use it, that does not sound very user friendly.

@tniessen I thought about it again and what about a very simple solution: adding a small user notification that the test is run with flags. That would a) remove the "magic" part as it's now clear what happens and b) everyone who forgot to add some flags while debugging native code would also realize that it's a different process.

[tniessen]

@BridgeAR I was about to propose a process warning but thought that might interfere with some tests. I like your idea!

[BridgeAR]

I added a comment. PTAL.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/21963/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/21983/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/22001/ ✔️

[BridgeAR]

Landed in f2dc99c and 6e5ffc4 🎉

Thanks for the reviews!