-
Notifications
You must be signed in to change notification settings - Fork 227
Rewrite testing with CMake's CTest #646
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit rewrites the tests in this repository to using CMake's testing harness CTest. This is borne out of current and historical frustrations I've had with testing in this repository with the goal of laying a solid foundation for future growth with testing. Some specific rationale for this change is: * Current tests cannot be run in parallel. This means that tests also cannot be built in parallel since it's bundled together right now. The reason for this is due to the introduction of sockets-related tests that all use the same port. CTest has the ability to specify that a set of tests all lock an abstract "resource" which means that they run serially, solving this issue. * Debugging test failures is difficult right now where a test can be reported as failed but a file must be consulted to see the output of the test. CTest has robust facilities and flags for capturing test output and reporting it, solving this issue. * Running one test right now is not easy to do. A test can be run but whether it passed/failed needs to be consulted via an output file. The `ctest` command line has baked-in abilities to run one or a subset of tests. * Adding a timeout to tests (often helpful for networking-related tests) is not currently possible. While enough shell-script-magic might make this work it's much easier to use CTest's built-in timeout mechanism. * Current shell-script-based-testing has no means of testing what the output of a test is, but CTest has this built in. This means that there's now a test that prints hello world and will fail if nothing is printed. * The makefile-based tests currently print no indication of what test is running. With sockets-tests often taking 1s+ each it means that a test run can seemingly hang while it's actually running tests. CTest has lots of built in options and a reasonable default for displaying test progress. * Per-test configuration is done currently in the test file itself which is parsed out via Python scripts. While this works well enough it's project-specific and can be difficult to remember syntax. CMake and CTest combined have many options of how to configure tests in the same manner. A downside of CMake/CTest, however, is that test configuration is separate from the test itself (e.g. in `CMakeLists.txt` as opposed to the test). * In broad strokes I expect CMake/CTest to be more robust about avoiding using stale artifacts or reporting on stale results. I got myself into weird situations with the makefile-based tests where the only way I could trust the output was to blow away the entire test build directory between each run. CTest/CMake are more often than not "right by default" in terms of build dependencies unlike makefiles where everything needs to be built from scratch. * Personally I find shell scripts very hard to parse and read. The current `run-test.sh` is a large wad of shell and is difficult to debug and/or run tests in isolation. With CTest all handling of processes and such is handled by CTest itself and a verbose command line option (`-V`) can print the exact command per-test which means that Wasmtime, for example, can be directly executed for tests. Overall I personally feel that CTest is a much more solid foundation on which to build tests in wasi-libc. It provides many more features by-default than building a custom test harness. While it's possible to build makefiles that handle all of the above cases better I don't think we have the development resources in wasi-libc to maintain such a test harness and suite. Instead I feel it's better to hitch the wagon to an externally-maintained solution with CTest and CMake. This commit does not transition wasi-libc's main build system to CMake and instead leaves the top-level `Makefile` untouched. That's left for another day. Some other minor changes in this are: * The `libc-test` dependency is now pinned to a commit * Component tests use `wasm-component-ld` instead of `wasm-tools` to build components. * Tests from `libc-test` no longer have a file in this repo with a `#include` of the source but are instead directly configured in CMake configuration. * Two sockets-related tests are disabled due to spurious failures found while testing locally. * The `test/Makefile`, `test/scripts/*.py`, and `run-test.sh` files are all deleted. * With `test/src/libc-test` now being gone all tests are moved from `test/src/misc` to `test/src`. * Some minor test adjustments were made which were found to be spurious failures while developing this. * Tests for `wasm32-wasi-threads` were deleted since that was actually testing `wasm32-wasip1-threads` and didn't provide any benefit over that test builder. cc WebAssembly#322 Closes WebAssembly#604 Closes WebAssembly#627
d8dcb95 to
898cd60
Compare
* Sleep based on a total amount of time instead of a factor of the resolution to make the test more deterministic and predictable. This also avoids `tv_nsec` overflowing if the resolution is too high. * Fix `nanoseconds_elapsed` calculation to multiple the seconds taken by the number 1B. * Adjust the `difference` calculation to use a variable directly instead of only part of a struct.
pchickey
approved these changes
Nov 10, 2025
Collaborator
pchickey
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not a CTest expert by any means but I trust that its a better mechanism than the Makefile and shell that was here previously, which I also found pretty difficult to understand.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit rewrites the tests in this repository to using CMake's testing harness CTest. This is borne out of current and historical frustrations I've had with testing in this repository with the goal of laying a solid foundation for future growth with testing. Some specific rationale for this change is:
Current tests cannot be run in parallel. This means that tests also cannot be built in parallel since it's bundled together right now. The reason for this is due to the introduction of sockets-related tests that all use the same port. CTest has the ability to specify that a set of tests all lock an abstract "resource" which means that they run serially, solving this issue.
Debugging test failures is difficult right now where a test can be reported as failed but a file must be consulted to see the output of the test. CTest has robust facilities and flags for capturing test output and reporting it, solving this issue.
Running one test right now is not easy to do. A test can be run but whether it passed/failed needs to be consulted via an output file. The
ctestcommand line has baked-in abilities to run one or a subset of tests.Adding a timeout to tests (often helpful for networking-related tests) is not currently possible. While enough shell-script-magic might make this work it's much easier to use CTest's built-in timeout mechanism.
Current shell-script-based-testing has no means of testing what the output of a test is, but CTest has this built in. This means that there's now a test that prints hello world and will fail if nothing is printed.
The makefile-based tests currently print no indication of what test is running. With sockets-tests often taking 1s+ each it means that a test run can seemingly hang while it's actually running tests. CTest has lots of built in options and a reasonable default for displaying test progress.
Per-test configuration is done currently in the test file itself which is parsed out via Python scripts. While this works well enough it's project-specific and can be difficult to remember syntax. CMake and CTest combined have many options of how to configure tests in the same manner. A downside of CMake/CTest, however, is that test configuration is separate from the test itself (e.g. in
CMakeLists.txtas opposed to the test).In broad strokes I expect CMake/CTest to be more robust about avoiding using stale artifacts or reporting on stale results. I got myself into weird situations with the makefile-based tests where the only way I could trust the output was to blow away the entire test build directory between each run. CTest/CMake are more often than not "right by default" in terms of build dependencies unlike makefiles where everything needs to be built from scratch.
Personally I find shell scripts very hard to parse and read. The current
run-test.shis a large wad of shell and is difficult to debug and/or run tests in isolation. With CTest all handling of processes and such is handled by CTest itself and a verbose command line option (-V) can print the exact command per-test which means that Wasmtime, for example, can be directly executed for tests.Overall I personally feel that CTest is a much more solid foundation on which to build tests in wasi-libc. It provides many more features by-default than building a custom test harness. While it's possible to build makefiles that handle all of the above cases better I don't think we have the development resources in wasi-libc to maintain such a test harness and suite. Instead I feel it's better to hitch the wagon to an externally-maintained solution with CTest and CMake.
This commit does not transition wasi-libc's main build system to CMake and instead leaves the top-level
Makefileuntouched. That's left for another day. Some other minor changes in this are:libc-testdependency is now pinned to a commitwasm-component-ldinstead ofwasm-toolsto build components.libc-testno longer have a file in this repo with a#includeof the source but are instead directly configured in CMake configuration.test/Makefile,test/scripts/*.py, andrun-test.shfiles are all deleted.test/src/libc-testnow being gone all tests are moved fromtest/src/misctotest/src.wasm32-wasi-threadswere deleted since that was actually testingwasm32-wasip1-threadsand didn't provide any benefit over that test builder.cc #322
Closes #604
Closes #627