Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

recheck always fails #408

Closed
dharmaturtle opened this issue Apr 24, 2022 · 5 comments · Fixed by #415
Closed

recheck always fails #408

dharmaturtle opened this issue Apr 24, 2022 · 5 comments · Fixed by #415

Comments

@dharmaturtle
Copy link
Member

I have the following test:

    testCase "my test" <| fun () ->
        property {
            let! i = Gen.int32 (Range.linear 1 10)
            Expect.isTrue (i <= 5)
        }
        |> Property.check

It fails (as expected) with the following error:

    *** Failed! Falsifiable (after 60 tests):
    6
    Expecto.AssertException: Should be true. Actual value was false but had expected it to be true.
       at Expecto.Expect.isTrue@397.Invoke(String msg)
       at Expecto.Expect.isTrue(Boolean actual, String message)
       at Hedgehog.Tests.TestDsl.Expect.isTrue(Boolean value) in C:\Code\fsharp-hedgehog\tests\Hedgehog.Tests\TestDsl.fs:line 38
       at Hedgehog.Tests.PropertyTests.propertyTests@103-30.Invoke(Int32 _arg8) in C:\Code\fsharp-hedgehog\tests\Hedgehog.Tests\PropertyTests.fs:line 103
       at Hedgehog.Property.kTry@88.Invoke(a a) in C:\Code\fsharp-hedgehog\src\Hedgehog\Property.fs:line 89
    This failure can be reproduced by running:
    > Property.recheck "59_15683818216235017248_6890634259096173777_" <property>

I plug the suggested recheck into the test:

    testCase "my test" <| fun () ->
        property {
            let! i = Gen.int32 (Range.linear 1 10)
            Expect.isTrue (i <= 5)
        }
        |> Property.recheck "59_15683818216235017248_6890634259096173777_"

It fails with


    *** Failed! Falsifiable (after 1 test):
    6
    Expecto.AssertException: Should be true. Actual value was false but had expected it to be true.
       at Expecto.Expect.isTrue@397.Invoke(String msg)
       at Expecto.Expect.isTrue(Boolean actual, String message)
       at Hedgehog.Tests.TestDsl.Expect.isTrue(Boolean value) in C:\Code\fsharp-hedgehog\tests\Hedgehog.Tests\TestDsl.fs:line 38
       at Hedgehog.Tests.PropertyTests.propertyTests@103-30.Invoke(Int32 _arg8) in C:\Code\fsharp-hedgehog\tests\Hedgehog.Tests\PropertyTests.fs:line 103
       at Hedgehog.Property.kTry@88.Invoke(a a) in C:\Code\fsharp-hedgehog\src\Hedgehog\Property.fs:line 89

Great! Let's now make the test pass (trivially):

    testCase "my test" <| fun () ->
        property {
            let! i = Gen.int32 (Range.linear 1 10)
            Expect.isTrue true
        }
        |> Property.recheck "59_15683818216235017248_6890634259096173777_"

But it doesn't pass!

    *** Failed! Falsifiable (after 1 test):
    6

Is this intended behavior?

@TysonMN
Copy link
Member

TysonMN commented Apr 25, 2022

No. The actual behavior is not the intended behavior. The behavior you expected in the intended behavior.

I have a guess about the cause of the bug. When rechecking, I might have assumed failure. Clearly I should not be assuming that.

@AlexeyRaga
Copy link

Bump!
I stumbled upon the same issue where recheck always fails. Which is extremely misleading, and also brings a lot of frustration to my colleagues adopting property-based testing and Hedgehog.

@TysonMN
Copy link
Member

TysonMN commented Jul 21, 2022

Sorry! I will fix this now.

TysonMN added a commit to TysonMN/fsharp-hedgehog that referenced this issue Jul 21, 2022
TysonMN added a commit to TysonMN/fsharp-hedgehog that referenced this issue Jul 21, 2022
TysonMN added a commit to TysonMN/fsharp-hedgehog that referenced this issue Jul 23, 2022
TysonMN added a commit to TysonMN/fsharp-hedgehog that referenced this issue Jul 23, 2022
TysonMN added a commit to TysonMN/fsharp-hedgehog that referenced this issue Jul 23, 2022
@TysonMN
Copy link
Member

TysonMN commented Jul 23, 2022

First included in version 0.13.0

@TysonMN
Copy link
Member

TysonMN commented Jul 23, 2022

Sorry everyone for how long I waited before fixing this bug. In hindsight, it was an easy fix. I was expecting it to be difficult, so I kept putting it off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants