Improve test harness and add browser pooling

[compulim]

Fix #3736.
Fix #3557.
Fix #3842.

Changelog Entry

Added

• Resolved #3557 and #3736. Improved test harness and added browser pooling, by @compulim, in PR #3871

Fixed

• Fixes #3842. Updated husky, lint-staged and corresponding precommit scripts, by @compulim, in PR #3871

Description

Goals of this PR:

• Improve DX on test development
  • Converged test behaviors between Chrome and Jest/WebDriver
    • Chrome can use sendKeys to send TAB and other special keys
    • This simplify debugging of tests with race conditions
  • Improved debuggability/runnability in Chrome
  • Auto restart Docker images on local changes
• Improve test speed
  • Pooling WebDriver session
  • Remove unnecessary Babel/React
• Improve test reliability
  • Wait for all images loaded before taking snapshot
• Improve dev scripts on husky, lint-staged and Git pre-commit hook

New "HTML" test harness

Today/yesterday, when we develop some HTML tests that requires special keyboard input (e.g. TAB), it can only be automated via Jest/WebDriver, but not while developing (in pure Chrome). To mitigate the issue, we log to console and ask the developer to do the input manually. This means, tests running in dev mode and Jest/CI could have different behaviors.

We added a new npm run browser command to start a Chrome that can be remotely controlled by a corresponding WebDriver client. It can send special keyboard inputs, emulated pointer input, and take screenshots.

Please download a matching version of ChromeDriver.exe and put it under the project root. If you are on WSL, you will need chmod +x ChromeDriver.exe to enable execute permission.

• Run npm run browser to start a browser session for HTML tests
  • Using this browser to load __tests__/*.html will use WebDriver to run tests on them
  • View the screenshot from console for easy debugging
• Run npm run docker to start a Docker Compose
  • This is only needed for Jest
  • Any files changed, will rebuild and restart of the compose automatically

Separate test harness and test page object model

Today/yesterday, test harness and page object model are mingled together. Clearer responsibilities on packages will increase DX.

It was packages/testharness, and now into 2 packages: packages/test/harness and packages/test/page-object-model.

• Test harness is the infrastructure to run tests, it use WebDriver behind the scene
• Test page object model is the object model for controlling Web Chat, no knowledge of WebDriver is needed
• Both test harness and page object models does not use regenerator-runtime, give a much better error stack

Test code changed

• body element will have a class named jest, if the page is running under Jest (or CI)
  • Can be used to hide elements for test development
• await pageConditions.uiConnected(), instead of await host.wait(conditions.uiConnected(), 15000)
• Use run(async () => {}), instead of IIFE with host.done() or host.error()
• No Babel or React is loaded by default, should be quicker
• Standardized all xxxStabilized code by leveraging the new pageConditions.stabilized()
  • See scrollStabilized() or the advanced scrollToBottomCompleted() for usage

Browser session pooling

Today/yesterday, browser sessions are disposed immediately after tests. It takes about 1 second to launch a new browser session. With ~240 HTML tests, it took 4 extra minutes.

This feature reduce that 4 minutes to (240 / 5) * 1s ~= 48 * 1s = 48 seconds (-192 seconds).

Added pooling by a WebDriver proxy, given capabilities are unchanged.

• Browser sessions are now pooled with max 5 runs and 3 minutes (the last 30 seconds are not acquirable)
• Saved about 1 second per test run that can reuse sessions, or about 3 minutes
• Increase memory from 2 GB to 2.5 GB for each session for stability
• Jest only parallelize test suites, but not tests inside each suite
• Few things to reset:
  1. Navigate to about:blank
  2. Move mouse to (0, 0)
  3. Reset window size to 360x640

Design

The "WebDriver session pooling server" proxy will intercepts new session and delete session requests.

We move window resizing from capabilities to runtime. This means all tests no longer have unique capabilities, this enable simple session pooling.

Every instance in the pool will survive 5 runs, or 3 minutes maximum. We assume each session will take up to 30 seconds. That means, after hitting 2:30 minutes, the instance will no longer to acquire. If it was acquired, it will be continue available until release.

Specific Changes

(Other than specified above)

• Rename Dockerfile-xxx to xxx.dockerfile for VSCode syntax highlighting
• Updated npm run docker to pick up changes and restart images automatically
• Separated test harness and test page object model packages
• Update some snapshots
  • Button click in the new test harness will also move the mouse (to mimic a real mouse behavior)
    • Thus, the snapshot will show hover effect on the button
  • Some test reliabilities
• Cleaner HTML test files
  • Less one-time imports
• Don't load React/Babel in test harness, HTML file need to specify them if needed
  • Not every test need JSX, not loading React/Babel will speed up tests
• No more pre-build for testharness, means, npm install or npm run bootstrap will be faster

• I have added tests and executed them locally
• I have updated CHANGELOG.md
• I have updated documentation (Will do in another PR)

Review Checklist

This section is for contributors to review your work.

• Accessibility reviewed (tab order, content readability, alt text, color contrast)
• Browser and platform compatibilities reviewed
• CSS styles reviewed (minimal rules, no z-index)
• Documents reviewed (docs, samples, live demo)
• Internationalization reviewed (strings, unit formatting)
• package.json and package-lock.json reviewed
• Security reviewed (no data URIs, check for nonce leak)
• Tests reviewed (coverage, legitimacy)

[compulim]

Commented.

[compulim]

3 things to discuss offline. I have resolved all comments. Please let me know if I overlooked anything. 😉

[corinagum]

LGTM - don't see a need to wait for our 1:1 to approve :)

[corinagum]

Looks like the failing test is an unintended bug:

(non-interactive card should not be focusable)

I think this test is incorrectly named - behavior looks correct to me, considering there are inputs within the card.