Partial fix for upcoming testthat release

[hadley]

expect_success() and expect_failure() now test that you have exactly one success/failure and zero failures/successes. I can't quite figure out why the tests are still failing here; maybe it's something to do with recycling? I'm happy to help but I unfortunately I don't know enough about the lintr internals to figure out what's going wrong here.

We're planning to submit testthat to CRAN on Nov 10.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.24%. Comparing base (275ed7d) to head (8914fb0).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2937      +/-   ##
==========================================
- Coverage   99.24%   99.24%   -0.01%     
==========================================
  Files         129      129              
  Lines        7283     7274       -9     
==========================================
- Hits         7228     7219       -9     
  Misses         55       55              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[MichaelChirico]

Thanks @hadley! Can I check if this is back-compatible? We have testthat (>= 3.2.1),, I just confirmed it works on 3.2.3 at least

NVM, confirmed succeed() and fail() are present in 3.2.1:

https://github.com/cran/testthat/blob/0c2c0a2c32a8442a4c3a6f0333e22b12678af1a5/R/expect-that.R#L53-L61

[MichaelChirico]

I can't quite figure out why the tests are still failing here

AFAICT you were only failing the {lintr} check, the main thing was by switching from Map() to a nested for loop it increased the cyclomatic complexity of the implementation enough to trigger a lint. I refactored the nested loops into a helper instead. Bonus -- we could drop the itr_env (replacing older <<- usage) in the new approach which I also find cleaner.

Thanks for the proactive fix! Your release time is perfectly fine. We are overdue for a release anyway.

[hadley]

Since I submitted this, we realised that we can improve the description of expectations a bit more: https://testthat.r-lib.org/dev/articles/custom-expectation.html#expectation-basics. The key idea is now that your expectation should always return its first input invisibly, regardess of whether it succeeds or fails. I think this probably doesn't apply to lintr (since it's not the sort of expectation that you're likely to chain), but I thought you should know since you're one of the few people who have written a custom expectation AND have seen or interim advice.

[MichaelChirico]

For posterity:

The issue here is that fail() doesn't interrupt execution, and essentially we have:

myExpectation <- function(x) {
  maybeFail()
  succeed()
}

maybeFail <- function(x) {
  if (x < 0) fail('negative')
}

expect_failure(myExpectation(-1))

So as written (and my own guess) we might think "once fail() is reached, succeed() can't be" but that's not true.

[hadley]

Do you think that this could be clearer in https://testthat.r-lib.org/articles/custom-expectation.html ? Maybe I need to say something like this ?

It's easy to think that fail() is similar to stop() or abort(), because that's how it behaves when you run it interactively in the console. But it can't work this way in tests because tests need to keep running even after failure so that they can report all the failures at once. So in a real testing setup, fail() and pass() work very similarly: they both signal a condition that testhat records and then reports to the user.

[MichaelChirico]

Yes that's quite helpful! The examples in the vignette hint at this by always having if/else structure to pass/fail . But as you mentioned, it's deceptively similar to stop/abort, so useful to call out the difference.