Tracking features to support the runtime test-suite

[KaruroChori]

TODO:

• Implement VS_HEADLESS flag
• Implement VS_TEST to enable the internal testing interface
• Implement the internal headless operation interface (triggering test events and forcing events on ui-tree elements)
• Implement VS_LOG_FILE to define where to place the output from global::debug
• Implement global::debug
  • In C script
  • In JS script
  • From global context
  • From C bindings
• Add special test function in scripts to be called on special event test
  • In C script
  • In JS script
  • Support on ui-tree to scan and run all tests.
• Documentation

[andy5995]

Do you plan to use Xvfb for testing or do you have something else in mind?

I see it's used by libfltk but only to get a screenshot

subprojects/libfltk/.github/workflows/build_fluid_docs.yml
30: sudo apt-get install -y xvfb
36:## xvfb creates a headless X server for us, so we can render snapshot with FLUID
66: Xvfb :19 -screen 0 1024x768x16 &

[KaruroChori]

Short answer: possibly, but it is quite low on the priority list for now, unless you have a case in favour.

For context, there are several types of tests I would like to run:

• Tests to verify the build process has no regression on all supported architectures.
• Simple tests (and benchmarks) of features, like the name resolution, or the caching mechanisms, so to ensure specific functionalities of the vs-fltk library are properly implemented and free from memory leaks.
• Partial runtime tests (and benchmarks), where the vs is run in headless mode (basically the FLTK run function is never called). Debug entries are collected in a csv, and the test function for each embedded script is run. Also, a set of actions is run to simulate user interaction. Some of these tests generates reports to track memory leaks & dynamic memory temporal plots.
• Full runtime tests running on wayland/x11, where screenshots are taken. Those are meant to identify visual/layout regression.

For now, only the first three types are a priority. There is no specific need for a different X server to run them, if not to avoid the "memory leaks" X11 has, which admittedly lessen the usability of valgrind's reports.

[andy5995]

My only first-hand experience with using Xvfb for testing is basically:

In a shell script, run with meson test: start Xvfb, start the graphical app (in my case, rmw displaying a curses-based menu), and then kill the app. If the app is built with sanitize, or running through valgrind, and leaks or other problems found, the test will fail. (Although honestly I don't know if it's the best way for this use-case).

[KaruroChori]

For (my future) reference, I tested Xvfb. Sadly those leaks I was talking about are caused by the font rendering library, not xorg and clearly not xvfb.
Still, it is a nice option for testing purposes, and I am sure it will have its own place in the testing pipeline.