test_runner: add snapshot testing #53169

cjihrig · 2024-05-26T18:16:31Z

test_runner: add context.fullName

This commit adds a fullName getter to the TestContext and
SuiteContext classes. This is similar to the existing name getter,
but also includes the name of all ancestor tests/suites.

doc: add context.assert docs

When context.assert was added, no docs were added. This commit
adds initial documentation for context.assert because the
snapshot() function requires them.

Refs: #52860

test_runner: add snapshot testing

This commit adds a t.assert.snapshot() method that implements
snapshot testing. Serialization uses JSON.stringify() by default,
but users can configure the serialization to meet their needs.

Fixes: #48260

nodejs-github-bot · 2024-05-26T18:16:36Z

Review requested:

@nodejs/test_runner

benjamingr · 2024-05-27T19:49:08Z

@targos

doc/api/test.md

lib/internal/test_runner/snapshot.js

test/parallel/test-runner-snapshot-tests.js

benjamingr

Nice work, good tests, left some comments, want to see how people feel about this but personally I'm +1

JakobJingleheimer · 2024-05-27T21:42:23Z

I believe snapshot does not belong in core because it's too specific to a domain.

Chamindu36 · 2024-05-28T10:35:43Z

doc/api/test.md

+exports[`suite of snapshot tests > snapshot test 1`] = `
+{
+  "value1": 1,
+  "value2": 2


@cjihrig It seems like this is missing backtick consistency for string values in snapshots.

I'm not sure what you mean.

cjihrig · 2024-05-28T16:38:06Z

I believe snapshot does not belong in core because it's too specific to a domain.

I think this is a little too vague. What domain? What can't we do that most other test runners have figured out how to do?

nodejs-github-bot · 2024-05-28T16:38:53Z

CI: https://ci.nodejs.org/job/node-test-pull-request/59493/

doc/api/test.md

lib/internal/test_runner/test.js

lib/internal/test_runner/snapshot.js

Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) #53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) #53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) #53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) #51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) #52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) #52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) #53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) #52848 PR-URL: #53379

Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) #53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) #53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) #53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) #51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) #52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) #52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) #53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) #53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) #52848 PR-URL: #53379

DylanPiercey · 2024-06-11T21:23:02Z

@cjihrig is there any reason JSON.stringify was chosen here over util.inspect? The inspect api supports a bunch more data types, circular properties, and allows defining custom overrides.

Seems like it'd be a good fit here?

cjihrig · 2024-06-11T21:25:49Z

Yes. Per the util.inspect() docs:

The output of util.inspect may change at any time and should not be depended upon programmatically.

It's not a suitable default for Node core, but you can use it for your own serialization if you want.

DylanPiercey · 2024-06-11T21:28:45Z

Ah, lame. Alright, thanks for the heads up!

It'd be nice if the stance was changed on that api to only change output across majors. It seems reasonable to me that you might have to update your snapshots when upgrading to a new major. Oh well!

cjihrig · 2024-06-11T21:46:21Z

I would also love to see someone get a better and reliable serialization format in place!

benjamingr · 2024-06-11T22:05:53Z

util.inspect was actually used in the past iteration of snapshot testing we had and it caused problems people complained about (about how it defines equality and semverness)

This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: nodejs#53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>

When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: nodejs#53169 Refs: nodejs#52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

This commit adds a t.assert.snapshot() method that implements snapshot testing. Serialization uses JSON.stringify() by default, but users can configure the serialization to meet their needs. PR-URL: nodejs#53169 Fixes: nodejs#48260 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) nodejs#53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) nodejs#53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) nodejs#53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) nodejs#52762 * add StefanStojanovic to collaborators (StefanStojanovic) nodejs#53118 * add Marco Ippolito to TSC (Rafael Gonzaga) nodejs#53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) nodejs#53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) nodejs#51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) nodejs#52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) nodejs#52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) nodejs#53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) nodejs#52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) nodejs#53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) nodejs#53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) nodejs#52848 PR-URL: nodejs#53379

This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: nodejs#53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>

When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: nodejs#53169 Refs: nodejs#52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

This commit adds a t.assert.snapshot() method that implements snapshot testing. Serialization uses JSON.stringify() by default, but users can configure the serialization to meet their needs. PR-URL: nodejs#53169 Fixes: nodejs#48260 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) nodejs#53221 cli: * (SEMVER-MINOR) add `NODE_RUN_PACKAGE_JSON_PATH` env (Yagiz Nizipli) nodejs#53058 * (SEMVER-MINOR) add `NODE_RUN_SCRIPT_NAME` env to `node --run` (Yagiz Nizipli) nodejs#53032 doc: * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) nodejs#52762 * add StefanStojanovic to collaborators (StefanStojanovic) nodejs#53118 * add Marco Ippolito to TSC (Rafael Gonzaga) nodejs#53008 fs: * mark recursive cp methods as stable (Théo LUDWIG) nodejs#53127 lib: * (SEMVER-MINOR) add EventSource Client (Aras Abbasi) nodejs#51575 * (SEMVER-MINOR) replace MessageEvent with undici's (Matthew Aitken) nodejs#52370 module: * (SEMVER-MINOR) print amount of load time of a cjs module (Vinicius Lourenço) nodejs#52213 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) nodejs#53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) nodejs#52762 src: * (SEMVER-MINOR) traverse parent folders while running `--run` (Yagiz Nizipli) nodejs#53154 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) nodejs#53124 test_runner: * (SEMVER-MINOR) add snapshot testing (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) add context.fullName (Colin Ihrig) nodejs#53169 * (SEMVER-MINOR) support module mocking (Colin Ihrig) nodejs#52848 PR-URL: nodejs#53379

ChALkeR · 2024-07-13T17:12:07Z

Looking at that code, this method appears duplicated:

node/lib/internal/test_runner/test.js

Lines 1236 to 1244 in e08a654

    
           function getFullName(test) { 
        
             let fullName = test.name; 
        
             for (let t = test.parent; t !== t.root; t = t.parent) { 
        
               fullName = `${t.name} > ${fullName}`; 
        
             } 
        
             return fullName; 
        
           }

node/lib/internal/test_runner/test.js

Lines 228 to 230 in e08a654

    
           get fullName() { 
        
             return getFullName(this.#test); 
        
           }

Could be simplified with

node/lib/internal/test_runner/test.js

Lines 546 to 550 in e08a654

    
           getTestNameWithAncestors() { 
        
             if (!this.parent) return ''; 
        
             return `${this.parent.getTestNameWithAncestors()} ${this.name}`; 
        
           }

I.e. the test could have just had a .getPath which was joined with different strings likely? Perhaps I'll recheck

This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: #53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>

When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: #53169 Refs: #52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

This commit adds a fullName getter to the TestContext and SuiteContext classes. This is similar to the existing name getter, but also includes the name of all ancestor tests/suites. PR-URL: #53169 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com> Co-authored-by: Jacob Smith <jacob@frende.me>

When context.assert was added, no docs were added. This commit adds initial documentation for context.assert because the snapshot() function requires them. PR-URL: #53169 Refs: #52860 Reviewed-By: Moshe Atlow <moshe@atlow.co.il> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Geoffrey Booth <webadmin@geoffreybooth.com>

Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: TODO

Notable changes: buffer: * (SEMVER-MINOR) add .bytes() method to Blob (Matthew Aitken) #53221 doc: * doc-only deprecate OpenSSL engine-based APIs (Richard Lau) #53329 * (SEMVER-MINOR) add context.assert docs (Colin Ihrig) #53169 * (SEMVER-MINOR) improve explanation about built-in modules (Joyee Cheung) #52762 * add StefanStojanovic to collaborators (StefanStojanovic) #53118 * add Marco Ippolito to TSC (Rafael Gonzaga) #53008 inspector: * fix disable async hooks on Debugger.setAsyncCallStackDepth (Joyee Cheung) #53473 net: * (SEMVER-MINOR) add new net.server.listen tracing channel (Paolo Insogna) #53136 process: * (SEMVER-MINOR) add process.getBuiltinModule(id) (Joyee Cheung) #52762 src,permission: * (SEMVER-MINOR) --allow-wasi & prevent WASI exec (Rafael Gonzaga) #53124 test_runner: * (SEMVER-MINOR) add context.fullName (Colin Ihrig) #53169 util: * (SEMVER-MINOR) support `--no-` for argument with boolean type for parseArgs (Zhenwei Jin) #53107 PR-URL: #53945

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels May 26, 2024

cjihrig added semver-minor PRs that contain new features and should be released in the next minor version. commit-queue-rebase Add this label to allow the Commit Queue to land a PR in several commits. labels May 27, 2024

cjihrig force-pushed the snapshot branch from 67a15b1 to 3c76dbb Compare May 27, 2024 19:45

benjamingr reviewed May 27, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

benjamingr reviewed May 27, 2024

View reviewed changes

doc/api/test.md Show resolved Hide resolved

benjamingr reviewed May 27, 2024

View reviewed changes

lib/internal/test_runner/snapshot.js Show resolved Hide resolved

benjamingr reviewed May 27, 2024

View reviewed changes

lib/internal/test_runner/snapshot.js Show resolved Hide resolved

benjamingr reviewed May 27, 2024

View reviewed changes

test/parallel/test-runner-snapshot-tests.js Show resolved Hide resolved

benjamingr reviewed May 27, 2024

View reviewed changes

cjihrig mentioned this pull request May 27, 2024

test: expose file & fullName props on TestContext (for snapshots) #53179

Closed

Chamindu36 reviewed May 28, 2024

View reviewed changes

cjihrig force-pushed the snapshot branch from 3c76dbb to b0d70bb Compare May 28, 2024 16:36