-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Description
I propose two enhancements that, together, are aimed at helping developers keep total unit test times below an acceptable threshold.
The first enhancement is to have each unit test have a time limit. This would default to 1 second, and any test could change its own time limit with API such as std.testing.setCurrentTimeLimit(2.0)
. When running zig test
or zig build
there would be a flag to scale this number. For example, --test-timeout-scale=2.0
would give twice as much time to each unit test. Idea here is that you would set this value on hosts significantly slower or faster than normal.
When the time limit of a unit test is exceeded, the build system will kill the misbehaving test and report it as a timeout. zig test
would run forever and then report a timeout if it took longer than expected.
The second enhancement is a flag for discovering the relative durations of unit tests. Something like --test-timeout-diagnostics
which would display a sorted list of unit tests by how long they took to run.
When this flag is run at the build system level, it would aggregate the data across all sets of unit tests.
This is just a fun side benefit, but --test-timeout-diagnostics
is what suggestion would be given to folks asking "how do I disable the cache when running tests?" which would encourage, let's be honest, people who are probably doing naughty things in their unit tests, to at least think about how long they are taking to run.
When a child process times out and needs to be terminated, the parent process should first attempt to send a signal, allowing the child to dump a stack trace before terminating.