Skip to content

Commit

Permalink
feat: let should_fail_with check that the failure reason contains t…
Browse files Browse the repository at this point in the history
…he expected message (#5319)

# Description

## Problem

Resolves #4786

## Summary

`#[test(should_fail_with = "message")]` will now check that "message" is
a substring of the failure reason. I _think_ this is a
backwards-compatible change.

I thought about supporting regular expressions, as suggested in the
related issue, but I didn't know how to signal that it's a regex or just
"contains". I guess that could be done with another name, something like
`should_with_with_regexp` 🤔 (in a separate PR, if really needed/wanted)

## Additional Context

None

## Documentation

Check one:
- [ ] No documentation needed.
- [x] Documentation included in this PR.
- [ ] **[For Experimental Features]** Documentation to be submitted in a
separate PR.

# PR Checklist\*

- [x] I have tested the changes locally.
- [x] I have formatted the changes with [Prettier](https://prettier.io/)
and/or `cargo fmt` on default settings.
  • Loading branch information
asterite authored Jun 25, 2024
1 parent 96ef87b commit cb9db55
Show file tree
Hide file tree
Showing 4 changed files with 36 additions and 14 deletions.
21 changes: 19 additions & 2 deletions docs/docs/tooling/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ fn test_add() {
}
```

You can be more specific and make it fail with a specific reason by using `should_fail_with = "<the reason for failure>`:
You can be more specific and make it fail with a specific reason by using `should_fail_with = "<the reason for failure>"`:

```rust
fn main(african_swallow_avg_speed : Field) {
Expand All @@ -58,5 +58,22 @@ fn test_king_arthur() {
fn test_bridgekeeper() {
main(32);
}

```

The string given to `should_fail_with` doesn't need to exactly match the failure reason, it just needs to be a substring of it:

```rust
fn main(african_swallow_avg_speed : Field) {
assert(african_swallow_avg_speed == 65, "What is the airspeed velocity of an unladen swallow");
}

#[test]
fn test_king_arthur() {
main(65);
}

#[test(should_fail_with = "airspeed velocity")]
fn test_bridgekeeper() {
main(32);
}
```
17 changes: 6 additions & 11 deletions test_programs/noir_test_failure/should_fail_mismatch/src/main.nr
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,9 @@
fn test_different_string() {
assert_eq(0, 1, "Different string");
}
// The assert message has a space
#[test(should_fail_with = "Not equal")]
fn test_with_extra_space() {
assert_eq(0, 1, "Not equal ");
}
// The assert message has a space
#[test(should_fail_with = "Not equal")]
fn test_runtime_mismatch() {
// We use a pedersen commitment here so that the assertion failure is only known at runtime.
assert_eq(std::hash::pedersen_commitment([27]).x, 0, "Not equal ");
}

// The failure reason is a substring of the expected message, but it should be the other way around
#[test(should_fail_with = "Definitely Not equal!")]
fn test_wrong_expectation() {
assert_eq(0, 1, "Not equal");
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ fn test_should_fail_with_match() {
assert_eq(0, 1, "Not equal");
}

#[test(should_fail_with = "Not equal")]
fn test_should_fail_with_match_partial_match() {
assert_eq(0, 1, "Definitely Not equal!");
}

#[test(should_fail)]
fn test_should_fail_without_match() {
assert_eq(0, 1);
Expand Down Expand Up @@ -48,6 +53,11 @@ unconstrained fn unconstrained_test_should_fail_with_match() {
assert_eq(0, 1, "Not equal");
}

#[test(should_fail_with = "Not equal")]
unconstrained fn unconstrained_test_should_fail_with_match_partial_match() {
assert_eq(0, 1, "Definitely Not equal!");
}

#[test(should_fail)]
unconstrained fn unconstrained_test_should_fail_without_match() {
assert_eq(0, 1);
Expand Down
2 changes: 1 addition & 1 deletion tooling/nargo/src/ops/test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ fn check_expected_failure_message(
};

let expected_failure_message_matches =
matches!(&failed_assertion, Some(message) if message == expected_failure_message);
matches!(&failed_assertion, Some(message) if message.contains(expected_failure_message));
if expected_failure_message_matches {
return TestStatus::Pass;
}
Expand Down

0 comments on commit cb9db55

Please sign in to comment.