Framework: Speeding Up The Tests

[gziolo]

This issues replaces #2108 where we have collected what we found out on this topic.

Action items

During last Calypso hangout we agreed to the following action Items:

• look into switching test reporter (@blowery, @gziolo)
• look into parallelism (@blowery, @gziolo)
• @mtias will wrangle developers to help port and modify their tests to single runner
• write P2 to urge more developers to writing single test runners in higher level folders (@blowery)
• update docs with what they’ve learned so far - P2 mentioned in bullet point above covers also that (@blowery)
• @umurkontaci and @gziolo - check old PRs related to single runner

High level tasks

Other tasks we are working on in parallel:

• Framework: Enable Parallel Tests on Circle-CI (@blowery)
• Framework: Add a client-wide test suite (@blowery)
• Add rules for i18n linting + Replace i18nlint with custom ESLint rules (@aduth)
• Move test runner to test folder (@blowery)
• Move test helpers into test/test/helpers(@blowery)
• Extract auto from lib/react-test-env-setup/index.js into test/helpers/use-fake-dom (Related discussion) (@gziolo)
• Consider moving single runner from Makefile to package.json and add to Circle CI as npm test (@gziolo)
• Migrate tests from Makefile to single test runner - (@aduth, @blowery, @gziolo)
• Update all docs containing content related to unit tests
• Remove no longer needed code (tests.json configuration files, shell scripts, CircleCI setup) once all tests are migrated (@gziolo)
• Enable parallel execution of single test runner on all CircleCI containers (@gziolo)
• Make sure useFilesystemMocks and useMockery work properly together - we should avoid using useFilesystemMocks in new tests and we should be good :)
• Consider using tear-down quick mocks between tests (Related discuusion) We decided to take YAGNI approach, we are going to implement it when there is first use case

[gziolo]

check old PRs related to single runner

We are heading in the completely different direction, so it doesn't look like we can reuse old migration scripts. I'm leaving it to @umurkontaci to figure out :) We moved all unit tests manually that were simple enough to cover them with script. All remaining tests contain browser globals or local file mocks. They need to be moved with care by developers.

[gziolo]

We can consider it done, follow up tasks are listed in #4915.