Failure tagging #14
Labels
Milestone
Comments
silentbicycle
added a commit
that referenced
this issue
Aug 20, 2017
Previously, theft had no way of distinguishing failures, so failures
that occurred with fairly minimal input would tend to shadow failures
only discovered with more complex instances.
Now, if a failure is tagged (by calling `theft_tag_failure(t, ID)`),
then shrinking will ignore changes that lead to failures with different
tags. Untagged failures will still be used, because they may be
previously undiscovered.
Setting the failure tag to THEFT_FAILURE_TAG_NONE clears the tag,
so wrapping calls:
theft_tag_failure(t, TAG_CRASHED_INSIDE_FUNCTION_some_fun);
enum some_fun_res res = some_fun(some, arguments);
theft_tag_failure(t, THEFT_FAILURE_TAG_NONE);
would tag failures caused by the wrapped function crashing.
Closes #14.
silentbicycle
added a commit
that referenced
this issue
Aug 20, 2017
Previously, theft had no way of distinguishing failures, so failures
that occurred with fairly minimal input would tend to shadow failures
only discovered with more complex instances.
Now, if a failure is tagged (by calling `theft_tag_failure(t, ID)`),
then shrinking will ignore changes that lead to failures with different
tags. Untagged failures will still be used, because they may be
previously undiscovered.
Setting the failure tag to THEFT_FAILURE_TAG_NONE clears the tag,
so wrapping calls:
theft_tag_failure(t, TAG_CRASHED_INSIDE_FUNCTION_some_fun);
enum some_fun_res res = some_fun(some, arguments);
theft_tag_failure(t, THEFT_FAILURE_TAG_NONE);
would tag failures caused by the wrapped function crashing.
Closes #14.
silentbicycle
added a commit
that referenced
this issue
Aug 20, 2017
Previously, theft had no way of distinguishing failures, so failures
that occurred with fairly minimal input would tend to shadow failures
only discovered with more complex instances.
Now, if a failure is tagged (by calling `theft_tag_failure(t, ID)`),
then shrinking will ignore changes that lead to failures with different
tags. Untagged failures will still be used, because they may be
previously undiscovered.
Setting the failure tag to THEFT_FAILURE_TAG_NONE clears the tag,
so wrapping calls:
theft_tag_failure(t, TAG_CRASHED_INSIDE_FUNCTION_some_fun);
enum some_fun_res res = some_fun(some, arguments);
theft_tag_failure(t, THEFT_FAILURE_TAG_NONE);
would tag failures caused by the wrapped function crashing.
Closes #14.
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, theft has no way of telling failures apart, so failures appearing with very simple input tend to shadow other more obscure failures. With a bit more information, theft could restrict shrinking to a particular kind of failure, rather than reporting ones that are already known.
The API could look like:
void theft_tag_failure(struct theft *t, size_t fail_id);
The failure id would be passed to the shrinking hooks. A
fail_idof 0 (orTHEFT_TAG_NONE) would clear the failure tag, and any other number would refer to a test-specific failure type. This could be called immediately before returningTHEFT_TRIAL_FAIL, and/or wrapping calls that sometimes crash intag(ID), crashy_function, tag(NONE).Note: Because of COW, setting/unsetting the tag from a forked child process will probably require communicating over the pipe.
Either
shrink_postcould returnSKIPbased on thefail_id(making restricting shrinking to a specific failure mode entirely the hooks' responsibility), or setting a failure tag could always restrict shrinking to that tag (no hook necessary), or restricting shrinking could be a config flag. This will need some experimentation.The text was updated successfully, but these errors were encountered: