Skip to content

Conversation

@alexcrichton
Copy link
Collaborator

This commit rewrites the tests in this repository to using CMake's testing harness CTest. This is borne out of current and historical frustrations I've had with testing in this repository with the goal of laying a solid foundation for future growth with testing. Some specific rationale for this change is:

  • Current tests cannot be run in parallel. This means that tests also cannot be built in parallel since it's bundled together right now. The reason for this is due to the introduction of sockets-related tests that all use the same port. CTest has the ability to specify that a set of tests all lock an abstract "resource" which means that they run serially, solving this issue.

  • Debugging test failures is difficult right now where a test can be reported as failed but a file must be consulted to see the output of the test. CTest has robust facilities and flags for capturing test output and reporting it, solving this issue.

  • Running one test right now is not easy to do. A test can be run but whether it passed/failed needs to be consulted via an output file. The ctest command line has baked-in abilities to run one or a subset of tests.

  • Adding a timeout to tests (often helpful for networking-related tests) is not currently possible. While enough shell-script-magic might make this work it's much easier to use CTest's built-in timeout mechanism.

  • Current shell-script-based-testing has no means of testing what the output of a test is, but CTest has this built in. This means that there's now a test that prints hello world and will fail if nothing is printed.

  • The makefile-based tests currently print no indication of what test is running. With sockets-tests often taking 1s+ each it means that a test run can seemingly hang while it's actually running tests. CTest has lots of built in options and a reasonable default for displaying test progress.

  • Per-test configuration is done currently in the test file itself which is parsed out via Python scripts. While this works well enough it's project-specific and can be difficult to remember syntax. CMake and CTest combined have many options of how to configure tests in the same manner. A downside of CMake/CTest, however, is that test configuration is separate from the test itself (e.g. in CMakeLists.txt as opposed to the test).

  • In broad strokes I expect CMake/CTest to be more robust about avoiding using stale artifacts or reporting on stale results. I got myself into weird situations with the makefile-based tests where the only way I could trust the output was to blow away the entire test build directory between each run. CTest/CMake are more often than not "right by default" in terms of build dependencies unlike makefiles where everything needs to be built from scratch.

  • Personally I find shell scripts very hard to parse and read. The current run-test.sh is a large wad of shell and is difficult to debug and/or run tests in isolation. With CTest all handling of processes and such is handled by CTest itself and a verbose command line option (-V) can print the exact command per-test which means that Wasmtime, for example, can be directly executed for tests.

Overall I personally feel that CTest is a much more solid foundation on which to build tests in wasi-libc. It provides many more features by-default than building a custom test harness. While it's possible to build makefiles that handle all of the above cases better I don't think we have the development resources in wasi-libc to maintain such a test harness and suite. Instead I feel it's better to hitch the wagon to an externally-maintained solution with CTest and CMake.

This commit does not transition wasi-libc's main build system to CMake and instead leaves the top-level Makefile untouched. That's left for another day. Some other minor changes in this are:

  • The libc-test dependency is now pinned to a commit
  • Component tests use wasm-component-ld instead of wasm-tools to build components.
  • Tests from libc-test no longer have a file in this repo with a #include of the source but are instead directly configured in CMake configuration.
  • Two sockets-related tests are disabled due to spurious failures found while testing locally.
  • The test/Makefile, test/scripts/*.py, and run-test.sh files are all deleted.
  • With test/src/libc-test now being gone all tests are moved from test/src/misc to test/src.
  • Some minor test adjustments were made which were found to be spurious failures while developing this.
  • Tests for wasm32-wasi-threads were deleted since that was actually testing wasm32-wasip1-threads and didn't provide any benefit over that test builder.

cc #322
Closes #604
Closes #627

This commit rewrites the tests in this repository to using CMake's
testing harness CTest. This is borne out of current and historical
frustrations I've had with testing in this repository with the goal of
laying a solid foundation for future growth with testing. Some specific
rationale for this change is:

* Current tests cannot be run in parallel. This means that tests also
  cannot be built in parallel since it's bundled together right now.
  The reason for this is due to the introduction of sockets-related
  tests that all use the same port. CTest has the ability to specify
  that a set of tests all lock an abstract "resource" which means that
  they run serially, solving this issue.

* Debugging test failures is difficult right now where a test can be
  reported as failed but a file must be consulted to see the output of
  the test. CTest has robust facilities and flags for capturing test
  output and reporting it, solving this issue.

* Running one test right now is not easy to do. A test can be run but
  whether it passed/failed needs to be consulted via an output file. The
  `ctest` command line has baked-in abilities to run one or a subset of
  tests.

* Adding a timeout to tests (often helpful for networking-related tests)
  is not currently possible. While enough shell-script-magic might make
  this work it's much easier to use CTest's built-in timeout mechanism.

* Current shell-script-based-testing has no means of testing what the
  output of a test is, but CTest has this built in. This means that
  there's now a test that prints hello world and will fail if nothing is
  printed.

* The makefile-based tests currently print no indication of what test is
  running. With sockets-tests often taking 1s+ each it means that a test
  run can seemingly hang while it's actually running tests. CTest has
  lots of built in options and a reasonable default for displaying test
  progress.

* Per-test configuration is done currently in the test file itself which
  is parsed out via Python scripts. While this works well enough it's
  project-specific and can be difficult to remember syntax. CMake and
  CTest combined have many options of how to configure tests in the same
  manner. A downside of CMake/CTest, however, is that test configuration
  is separate from the test itself (e.g. in `CMakeLists.txt` as opposed
  to the test).

* In broad strokes I expect CMake/CTest to be more robust about avoiding
  using stale artifacts or reporting on stale results. I got myself into
  weird situations with the makefile-based tests where the only way I
  could trust the output was to blow away the entire test build
  directory between each run. CTest/CMake are more often than not "right
  by default" in terms of build dependencies unlike makefiles where
  everything needs to be built from scratch.

* Personally I find shell scripts very hard to parse and read. The
  current `run-test.sh` is a large wad of shell and is difficult to
  debug and/or run tests in isolation. With CTest all handling of
  processes and such is handled by CTest itself and a verbose command
  line option (`-V`) can print the exact command per-test which means
  that Wasmtime, for example, can be directly executed for tests.

Overall I personally feel that CTest is a much more solid foundation on
which to build tests in wasi-libc. It provides many more features
by-default than building a custom test harness. While it's possible to
build makefiles that handle all of the above cases better I don't think
we have the development resources in wasi-libc to maintain such a test
harness and suite. Instead I feel it's better to hitch the wagon to an
externally-maintained solution with CTest and CMake.

This commit does not transition wasi-libc's main build system to CMake
and instead leaves the top-level `Makefile` untouched. That's left for
another day. Some other minor changes in this are:

* The `libc-test` dependency is now pinned to a commit
* Component tests use `wasm-component-ld` instead of `wasm-tools` to
  build components.
* Tests from `libc-test` no longer have a file in this repo with a
  `#include` of the source but are instead directly configured in CMake
  configuration.
* Two sockets-related tests are disabled due to spurious failures found
  while testing locally.
* The `test/Makefile`, `test/scripts/*.py`, and `run-test.sh` files are
  all deleted.
* With `test/src/libc-test` now being gone all tests are moved from
  `test/src/misc` to `test/src`.
* Some minor test adjustments were made which were found to be spurious
  failures while developing this.
* Tests for `wasm32-wasi-threads` were deleted since that was actually
  testing `wasm32-wasip1-threads` and didn't provide any benefit over
  that test builder.

cc WebAssembly#322
Closes WebAssembly#604
Closes WebAssembly#627
* Sleep based on a total amount of time instead of a factor of the
  resolution to make the test more deterministic and predictable. This
  also avoids `tv_nsec` overflowing if the resolution is too high.
* Fix `nanoseconds_elapsed` calculation to multiple the seconds taken by
  the number 1B.
* Adjust the `difference` calculation to use a variable directly instead
  of only part of a struct.
Copy link
Collaborator

@pchickey pchickey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a CTest expert by any means but I trust that its a better mechanism than the Makefile and shell that was here previously, which I also found pretty difficult to understand.

@alexcrichton alexcrichton merged commit 3084b68 into WebAssembly:main Nov 10, 2025
16 checks passed
@alexcrichton alexcrichton deleted the cmake-test branch November 10, 2025 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tests can't write to standard output streams build: Refactor how components are created/tested

2 participants