dot test-reporter includes more info

[romainmenke]

Version

22.2.0

Platform

all platforms

Subsystem

No response

What steps will reproduce the bug?

• run any test with node --test-reporter dot
• expect to see only dots and X
• see stack traces instead

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

See only dots and X

What do you see instead?

Stack traces and helpful debug info

Additional information

See : #51769

---

We rely on the previous way the dot reporter worked.
All the new output is actually breaking for us.

We picked the dot reporter for self tests on another test suite exactly because it omits debug info.

The lack of debug info and stack traces makes it a perfect smell test that runs the same on any platform.

Because this was changed in a minor node version we now have a broken main :D

I am unsure how to proceed.
Is there a way to restore the old behavior of the dot reporter?

[romainmenke]

I've internalized the previous version of the dot runner given how small and simple it was.
https://github.com/csstools/postcss-plugins/blob/main/packages/postcss-tape/test/_dot-reporter.cjs

Feel free to close this issue.

[RedYetiDev]

This behavior was changed in #52655, does that answer your question?

@nodejs/test_runner is there a way to revert the behavior via a CLI flag or similar?

[RedYetiDev]

Marking as feature-request, as it appears this is requesting a way to disable the new dot reporter.

[romainmenke]

Now that this is labeled as a feature request I should explain better why I saw value in the previous implementation of the dot reporter.

---

We author a lot of PostCSS plugins and also have our own test suite for PostCSS plugins.
This test suite is built on top of node:test.

To test this test suite itself we decided to use the dot reporter exactly because it only gives feedback on passes and failures.

This makes it easy to verify that failing tests fail in the expected way.
We are only interested in the sequence of passes and failures and the final exit code.

Information like stack traces is not helpful for our purposes because these can be different for each user and environment.

The lack of detailed information in the dot reporter also gave us the impression that it would be stable across node versions.

With the recent changes it is both unstable between node versions (because it is a new addition) and it is unstable between users.

This makes it impossible to record test runner output and compare new runs against previous recordings.

[marco-ippolito]

I would have approached it differently, having no stack trace by default (if you use dot you usually don't want the output to be verbose) and opt in with a flag --debug flag or --verbose (like phpunit).
Not sure if it's a good idea to change the behavior again, at this point I'd add a flag skip stack traces

[MoLow]

would have approached it differently, having no stack trace by default (if you use dot you usually don't want the output to be verbose) and opt in with a flag --debug flag or --verbose (like phpunit).
Not sure if it's a good idea to change the behavior again, at this point I'd add a flag skip stack traces

no reason for it to be a flag. it can be two separate reporters. the only question is if they should both reside in core, or should one be in user-land?
I am ok with having dot and dot-verbose reporters, or alternatively dot and dot-compact

[romainmenke]

Not sure if it's a good idea to change the behavior again

Changing it back has the benefit that it restores the previous behavior.
That makes it possible to test over multiple node versions (all the way back to node 18 if I recall correctly) and have consistent output.

Introducing a new dot-compact doesn't have this benefit.