Speed up the CKEditor 5 development environment

[pomek]

📝 Provide a description of the improvement

Why

• It's PITA that a quite simple change requires almost one hour to verify.
  • https://app.travis-ci.com/github/ckeditor/ckeditor5/builds/261295530
  • 5a9e7cb

What / ideas

• A new third thread could help. It could handle testing CKEditor 5 features.
• Try to execute CKEditor 5 features in parallel. Still determining how many threads are at the same time, but for sure, we can handle two processes simultaneously.
  • AFAIR Travis gives a single thread, so we must make some magic with promises and the JavaScript loop.
• Perhaps it's time to configure CircleCI to compare times.
• (?)

---

If you'd like to see this improvement implemented, add a 👍 reaction to this post.

[Reinmar]

• AFAIR Travis gives a single thread, so we must make some magic with promises and the JavaScript loop.

That won't speed up anything (at least not significantly) if it's still a single thread. Running two Chrome instances may also be prone to these focus issues unless the headless Chrome can take care of detaching focus/selection from the OS in which it operates.

[Reinmar]

It's PITA that a quite simple change requires almost one hour to verify.

48 minutes for automated tests and 25 minutes for other validation tasks. This indicates that we should indeed focus now on optimizing how automated tests are executed.

Here, the main thing to blame is that every package tests are run completely separately. The reason for this is that we wanted to have separate coverage for each of them. But this means that  testing every package start everything from scratch – build everything (transpile TS, run webpack, instrument coverage), start the env (Karma, Chrome), finally: execute the tests.

It'd be great to know which part of that process take how much time. Whether this is the building or execution step (starting the env is most likely fairly cheap). So, the imporant question to answer: What's percentage cost of the build and execution steps for a typical package?

If building is costly: Look for options to speed this up. We saw already how e.g. using Vite can help. Any progress here would also benefit our dev environment, so it'd be awesome to have meaningful improvements here, even if the build step is <40% of total time. Some ideas:

• Vite or other bundler?
• Caching between webpack processes?
• Prebuilding JS to TS once (similar to caching)?
• Using single build for all packages (which would required figuring out the magic to still count coverage separately for each package)?

If the execution of tests is the major problem, then we have fewer options, unfortunately.

• We might look for more powerful CI machines
• or split this task into two jobs

[filipsobol]

I agree that using promises will not help us much, but will probably increase the complexity of our tools. In the current state of the JavaScript ecosystem, we have to run TypeScript (tsc, ts-loader, etc.) to check types or generate d.ts files. However, for code transpilation and bundling we should aim to update our tools to something that can handle TypeScript files, but ignores typing, such as esbuild (I used it in the past and liked it a lot), vite (same), or swc.

I did some quick tests with promising results, but it (obviously) requires much more research.

I ran the yarn test -f XXX --reporter=dots --production --coverage command with the current Karma + webpack configuration for 3 packages and these are the results:

Package name	Total time	webpack time	webpack time as %
clipboard	36s	29s	81%
engine	71s	37s	52%
ui	44s	32s	73%

Then I replaced the ts-loader with esbuild-loader and ran the same commands:

Package name	Total time	webpack time	webpack time as %	Improvement over ts-loader
clipboard	10s	4s	40%	-72%
engine	42s	10s	19%	-41%
ui	17s	5s	29%	-61%

Then I did the same with swc-loader and results were identical. The only difference was that esbuild-loader had some minor issues with code coverage.

With this configuration, the travis-check.js script took 14m10s locally (down from 31m47s).

In my experience, if we went even further and replaced karma-webpack with karma-esbuild, then the build time would be negligible (200-1000ms). Unfortunately, to my knowledge switching to `esbuild` isn't something we can do quickly, because we have a lot of tools written specifically for webpack.

What we can probably do in a reasonable amount of time is to prebuild all the TypeScript code before running the tests, so that Webpack doesn't have to handle it. There are some improvements we can make in this area as well.

I ran build commands for all packages with the current setup, then updated tsconfigs to use TypeScript's project references (see changes on my test branch), then changed the same configuration to only emit declarations (no JS output).

Current setup	8m 02s
Project references	2m 25s
Project references + emitDeclarationOnly: true	2m 18s

Another small win might be the caching of yarn dependencies.

[pomek]

Thanks @filipsobol for the detailed research.

I am fine with replacing ts-loader with esbuild-loader. Then, we can talk about the next steps.

[przemyslaw-zan]

swc-loader seems to be quite problematic, as it causes many failed tests and coverage issues.

esbuild-loader caused less problems, as there are only minor gaps in code coverage, but the tests pass. Additionally, it is stricter than ts-loader, as case in which file exports a type but the index.ts of given package does not specify that export as a type, causes an error. [fix]

We need to find a solution to the broken coverage with the esbuild-loader. Swapping babel-loader to istanbul-instrumenter-loader does not help here.

[martnpaneq]

Some progress was achieved with esbuild-loader and CI is green now.

1. Most of the issues were caused by esbuild removing /* istanbul ignore */ comments when transpiling code from typescript to javascript. In order to keep them, -- @preserve option needs to be added to all of them.

2. In the engine package, there were some new errors, but their origin is not fully known.
   Looks like some paths, like missing else statements, that were previously just an info icon, are turning into errors - only on CI.

Prev:

Now:

To fix those errors I used 4 new /* istanbul ignore */ comments.

3. One /* istanbul ignore */ comment that was meant for a single case clause is not being preserved even with -- @preserve option (in this file).
   I tried writing a proper test for it, but it took too long time with no progress. I had to ignore the whole switch statement.

4. The last thing to fix are CK_DEBUG comments. If we want them preserved, we also need to add -- @preserve flag to them. I added some code to the loader that handles them in ckeditor5-dev, but I need some help with testing it.

There are 2 PRs to merge:
ckeditor/ckeditor5-dev#862 - needs testing for CK_DEBUIG loader with -- @preserve option
#13809 - needs to wait for the PR above and need to add-- @preserve option to CK_DEBUG comments