Migrate from Jest to Mocha

[connorjclark]

Motivation

Our usage of Jest has been identified as a blocker for migrating to ES modules. In short, there is a bug in V8 which we don't have a way to resolve, and even if we did, only the latest Node would be usable for our unit tests making migration impossible for some years. For more on that bug, see this and this.

Besides the ES modules story, we also wish to migrate away from Jest for other reasons–a common refrain is that tests written in Jest are verifiying that things work in a "Node + whatever Jest adds" framework, not just "Node". This may just be an act of idealism, as I don't recall running into anything significant differences in my esm migration branches that could be framed as "Jest not really testing Node", but it's one of those things you don't know until you explore it. At least without Jest, we can be more confident our tests aren't relying on an undocumented or not-well-understood artifact of its runtime.

Additionally, Jest transpiles all JS before running it, and the subtle transformations and indirection can create added debugging headaches.

New test runner

Mocha is a barebones test runner, so we'll need to add a few things on top to make a transition from Jest feasible. Capabilities Jest offers that we need to keep:

Expect

This is simple: although made by Jest, expect is a standalone npm package.

Snapshots

Jest implements snapshots with jest-snapshot, which is a standalone package. We can continue to utilize the same
snapshot tests and expectation files by utilizing this package in Mocha's setup module–adding the necessary extensions via expect.extend and calling the update logic in a root afterAll lifecycle function.

Mocks

In Jest, the term "mock" is a bit overloaded. The jest.fn(), jest.spyOn() is the mocking interface and acts on objects, and allows for expectations to be made on if/how often it was called. This is provided by jest-mock, and again: we can just use this package directly.

On the other hand, jest.mock acts on modules (give it a module path and Jest's runtime will replace it with the provided mock). To replace this functionality, we can use testdouble (jest.mock -> td.replace). It uses just Node's --loader API as opposed to Jest using V8's vm, which is where the fatal bug mentioned above lives.

Changing what mocking library we use (or even, removing mocks entirely) has not been evaluated, and is out of scope for this migration.

Timers

We make heavy use of Jest's fake timers... which again, is a standalone package we can just pull in: @jest/fake-timers.

Test isolation

By default, Mocha runs all tests in the same process. If given the flag --parallel, Mocha will spin up some worker processes which run a subset of each test (note: each process will still only run a single test at a time). Because there is no v8-level module isolation, we must ensure that tests don't impact the global state (or clean up when they do).

Unfortunately, pretty much every test that mocks modules, or uses fakes times, does some setup in the root test file scope (as opposed to a root test file beforeAll). Mocha processes will first import/require all the test files... and then start running each test file's suite one at a time. This means that the first thing that happens in Mocha is all of our root-level mocks / fake timer stuff is applied, greatly impacting the other tests that run in that process.

A simple approch to guarentee test isolation would be to run Mocha with one test at a time, and optionally manage a worker pool to approximate the same benefit of --parallel. However, we may have to pay noticeable overhead with Mocha and our setup file being repeated so much. Instead, we can run all the "non-mocks, non-timers" tests together in the same call to Mocha, and isolate just the problematic tests in separate calls to Mocha.

As a followup to this work, we can migrate each of these tests to do their setup in beforeAll. This is being deferred because there's a chance these changes could be subtle breakages, so best to do them in their own PR.

Should note: all the testdouble and fake timers state can be cleared after each test suite in a beforeAll root hook, defined in the Mocha setup file.

Similar CLI interface

With Jest's CLI, you can run a subset of test files by passing an argument that filters which tests files to run via a string-includes check: jest audits will run all the test files with "audits" in the name. To do the same with Mocha you need to use a glob: mocha 'lighthouse-core/**/audits/*'.

In Jest, to run test cases whose title matches a pattern, you do yarn jest -t pattern. With mocha, that is mocha --grep pattern.

These differences are enough to encourage a small wrapper around mocha to offer the same interface.

---

TLDR; Let's move away from Jest by using only parts of Jest. lol

Tasks

The bulk of this work will be done in one big PR (because keeping Jest and Mocha as test runners at the same time is a headache best avoided). However, there are some smaller bits of work that can be done first.

• Reach consensus on replacing Jest with Mocha
• tests: use readJson instead of imports for json #14020 Move away from non-standard Jest imports of json files
• tests(config-helpers): restore process.cwd after mocking #14036
• tests: replace jest with mocha #14054 Migrate all Jest tests to Mocha, using WIP: tests: convert core to mocha #14035 as a base

A number of draft PRs have been put up.

#13568 - This was a first pass, that converts just lighthouse-cli tests and attempts to keep Jest and Mocha around at the same time. Don't spend very long look at this, instead...
#14018 - This is an example PR of one of the simpler test folders to migrate
#14035 - This converts lighthouse-core tests, expanding on the mocha setup/wrapper script from the previous attempt, and does not try to keep Jest tests working. Areas of focus here are mocha-setup.cjs and run-tests.sh