fix(screenshot): alert user when toMatchScreenshot uses NaN

[rwaskiewicz]

What is the current behavior?

we have a few strictNullChecks violations for toMatchScreenshot, which are a result of a bug/not so-nice failure mode in the matcher.

GitHub Issue Number: N/A

What is the new behavior?

ensures that deviceScaleFactor is properly narrowed to type number from number | undefined. prior to this commit, the property could be set to undefined. this would result in multiplying by undefined, producing NaN in our calculation of mismatched pixels. this value would be used in a comparison operator, which would always return false.

that is, if deviceScaleFactor is undefined, we would always fail a test using this matcher. that logic doesn't actually change - instead, we throw a much more explicit error instead of silently failing

Does this introduce a breaking change?

• Yes
• No

Testing

Pull down this branch and build it. Make note of the location of the generated tarball:

npm run clean \                   
&& npm ci \
&& npm run build \
&& npm pack

Next, create a new stencil component library with that tarball:

cd /tmp \
&& npm init stencil@latest component jest-test \
&& cd $_ \
&& npm i [PATH_TO_TARBALL]

Navigate to src/components/my-component.e2e.ts in your new project and add the following tests:

  it('tests the matcher with empty compare object', async () => {
    expect({}).toMatchScreenshot();
  })

  it('tests the matcher with valid mismatchedPixels', async () => {
    expect({ mismatchedPixels: 123 }).toMatchScreenshot();
  })

  it('tests the matcher with valid mismatchedPixels and deviceScaleFactor', async () => {
    expect({ mismatchedPixels: 123, deviceScaleFactor: 123 }).toMatchScreenshot();
  })

  it('tests the matcher with valid mismatchedPixels, deviceScaleFactor and allowableMismatchedRatio', async () => {
    expect({ description: 'just a test', mismatchedPixels: 123, deviceScaleFactor: 123, allowableMismatchedRatio: 1, width: 1, height: 1 }).toMatchScreenshot();
  })

Run npm t -- --screenshot - tests fail as expected:

 FAIL  src/components/my-component/my-component.e2e.ts
  ● my-component › tests the matcher with empty compare object

    expect toMatchScreenshot() value is not a valid screenshot compare object - 'mismatchedPixels' has type undefined, but should be a number

      32 |
      33 |   it('tests the matcher with empty compare object', async () => {
    > 34 |     expect({}).toMatchScreenshot();
         |                ^
      35 |   })
      36 |
      37 |   it('tests the matcher with valid mismatchedPixels', async () => {

      at Object.toMatchScreenshot (node_modules/@stencil/core/testing/index.js:1655:52)
      at __EXTERNAL_MATCHER_TRAP__ (node_modules/expect/build/index.js:386:30)
      at Object.throwingMatcher [as toMatchScreenshot] (node_modules/expect/build/index.js:387:15)
      at Object.<anonymous> (src/components/my-component/my-component.e2e.ts:34:16)

  ● my-component › tests the matcher with valid mismatchedPixels

    expect toMatchScreenshot() value is not a valid screenshot compare object - 'deviceScaleFactor' has type undefined, but should be a number

      36 |
      37 |   it('tests the matcher with valid mismatchedPixels', async () => {
    > 38 |     expect({ mismatchedPixels: 123 }).toMatchScreenshot();
         |                                       ^
      39 |   })
      40 |
      41 |   it('tests the matcher with valid mismatchedPixels and deviceScaleFactor', async () => {

      at Object.toMatchScreenshot (node_modules/@stencil/core/testing/index.js:1656:53)
      at __EXTERNAL_MATCHER_TRAP__ (node_modules/expect/build/index.js:386:30)
      at Object.throwingMatcher [as toMatchScreenshot] (node_modules/expect/build/index.js:387:15)
      at Object.<anonymous> (src/components/my-component/my-component.e2e.ts:38:39)

  ● my-component › tests the matcher with valid mismatchedPixels and deviceScaleFactor

    expect toMatchScreenshot() missing allowableMismatchedPixels in testing config

      40 |
      41 |   it('tests the matcher with valid mismatchedPixels and deviceScaleFactor', async () => {
    > 42 |     expect({ mismatchedPixels: 123, deviceScaleFactor: 123 }).toMatchScreenshot();
         |                                                               ^
      43 |   })
      44 |
      45 |   it('tests the matcher with valid mismatchedPixels, deviceScaleFactor and allowableMismatchedRatio', async () => {

      at Object.toMatchScreenshot (node_modules/@stencil/core/testing/index.js:1684:9)
      at __EXTERNAL_MATCHER_TRAP__ (node_modules/expect/build/index.js:386:30)
      at Object.throwingMatcher [as toMatchScreenshot] (node_modules/expect/build/index.js:387:15)
      at Object.<anonymous> (src/components/my-component/my-component.e2e.ts:42:63)

Other information

[github-actions]

--strictNullChecks error report

Typechecking with --strictNullChecks resulted in 1418 errors on this branch.

That's 4 fewer than on main! 🎉🎉🎉

reports and statistics

Our most error-prone files

Path	Error Count
src/dev-server/index.ts	37
src/mock-doc/serialize-node.ts	36
src/dev-server/server-process.ts	32
src/compiler/build/build-stats.ts	27
src/compiler/output-targets/dist-lazy/generate-lazy-module.ts	25
src/compiler/style/test/optimize-css.spec.ts	23
src/testing/puppeteer/puppeteer-element.ts	23
src/compiler/prerender/prerender-main.ts	22
src/runtime/vdom/vdom-render.ts	20
src/runtime/client-hydrate.ts	19
src/screenshot/connector-base.ts	19
src/compiler/config/test/validate-paths.spec.ts	16
src/dev-server/request-handler.ts	15
src/compiler/prerender/prerender-optimize.ts	14
src/compiler/sys/stencil-sys.ts	14
src/compiler/transpile/transpile-module.ts	14
src/runtime/vdom/vdom-annotations.ts	14
src/sys/node/node-sys.ts	14
src/compiler/build/build-finish.ts	13
src/compiler/prerender/prerender-queue.ts	13

Our most common errors

Typescript Error Code	Count
TS2345	430
TS2322	406
TS18048	312
TS18047	100
TS2722	38
TS2532	36
TS2531	23
TS2454	14
TS2352	13
TS2769	10
TS2790	10
TS2538	8
TS2344	5
TS2416	4
TS2493	3
TS18046	2
TS2684	1
TS2488	1
TS2464	1
TS2430	1

Unused exports report

There are 12 unused exports on this PR. That's the same number of errors on main, so at least we're not creating new ones!

Unused exports

File	Line	Identifier
src/runtime/bootstrap-lazy.ts	21	setNonce
src/screenshot/screenshot-fs.ts	18	readScreenshotData
src/testing/testing-utils.ts	198	withSilentWarn
src/utils/index.ts	140	CUSTOM
src/compiler/app-core/app-data.ts	25	BUILD
src/compiler/app-core/app-data.ts	114	Env
src/compiler/app-core/app-data.ts	116	NAMESPACE
src/compiler/fs-watch/fs-watch-rebuild.ts	110	updateCacheFromRebuild
src/compiler/types/validate-primary-package-output-target.ts	62	satisfies
src/compiler/types/validate-primary-package-output-target.ts	62	Record
src/testing/puppeteer/puppeteer-declarations.ts	485	WaitForEventOptions
src/compiler/sys/fetch/write-fetch-success.ts	7	writeFetchSuccessSync

[rwaskiewicz]

This error message before wasn't the greatest - since I was already doing the same thing for deviceScaleFactor I decided to copy/paste/replace that error msg here too

[alicewriteswrongs]

The fix for this issue has been released as a part of today's Stencil v4.4.1 release.