Skip to content

Commit

Permalink
Stop early when coverage has been assured to be satisfied
Browse files Browse the repository at this point in the history
This also fixes an issue where tests would give false negatives when
coverage was below the specified percentages.

What was happening was that we would check if the result was verified -
it would say no and thus give us an incorrect error message:

"Insufficent coverage."
  • Loading branch information
felixmulder committed May 11, 2019
1 parent 8dbbdf0 commit e1d73b9
Showing 1 changed file with 6 additions and 4 deletions.
10 changes: 6 additions & 4 deletions hedgehog/src/Hedgehog/Internal/Runner.hs
Original file line number Diff line number Diff line change
Expand Up @@ -168,9 +168,7 @@ checkReport cfg size0 seed0 test0 updateUI =
-- If the user wants a statistically significant result, this function
-- will run a confidence check. Otherwise, it will default to checking
-- the percentage of encountered labels
maybe
(coverageSuccess count coverage)
(\c -> confidenceSuccess count c coverage) confidence
maybe False (\c -> confidenceSuccess count c coverage) confidence

failureVerified count coverage =
-- Will be true if we can statistically verify that our coverage was
Expand All @@ -191,6 +189,10 @@ checkReport cfg size0 seed0 test0 updateUI =
-- size has reached limit, reset to 0
loop tests discards 0 seed coverage0

else if tests > 0 && successVerified tests coverage0 then
-- tests have been verified to reach coverage for all labels
pure $ Report tests discards coverage0 OK

else if failureVerified tests coverage0 then
-- tests have been verified to not reach coverage for at least one label
pure . Report tests discards coverage0 . Failed $
Expand All @@ -206,7 +208,7 @@ checkReport cfg size0 seed0 test0 updateUI =

else if tests >= fromIntegral (propertyTestLimit cfg) then
-- we've hit the test limit
if successVerified tests coverage0 then
if coverageSuccess tests coverage0 then
-- all classifiers satisfied, test was successful
pure $ Report tests discards coverage0 OK

Expand Down

0 comments on commit e1d73b9

Please sign in to comment.