Skip to content

unit tests: enforce time limits #19821

@andrewrk

Description

@andrewrk

I propose two enhancements that, together, are aimed at helping developers keep total unit test times below an acceptable threshold.

The first enhancement is to have each unit test have a time limit. This would default to 1 second, and any test could change its own time limit with API such as std.testing.setCurrentTimeLimit(2.0). When running zig test or zig build there would be a flag to scale this number. For example, --test-timeout-scale=2.0 would give twice as much time to each unit test. Idea here is that you would set this value on hosts significantly slower or faster than normal.

When the time limit of a unit test is exceeded, the build system will kill the misbehaving test and report it as a timeout. zig test would run forever and then report a timeout if it took longer than expected.

The second enhancement is a flag for discovering the relative durations of unit tests. Something like --test-timeout-diagnostics which would display a sorted list of unit tests by how long they took to run.

When this flag is run at the build system level, it would aggregate the data across all sets of unit tests.

This is just a fun side benefit, but --test-timeout-diagnostics is what suggestion would be given to folks asking "how do I disable the cache when running tests?" which would encourage, let's be honest, people who are probably doing naughty things in their unit tests, to at least think about how long they are taking to run.

When a child process times out and needs to be terminated, the parent process should first attempt to send a signal, allowing the child to dump a stack trace before terminating.

Metadata

Metadata

Assignees

No one assigned

    Labels

    acceptedThis proposal is planned.proposalThis issue suggests modifications. If it also has the "accepted" label then it is planned.zig build systemstd.Build, the build runner, `zig build` subcommand, package management

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions