does this replace quickcheck completely?

[BurntSushi]

If the answer is yes, then I'd be happy to start directing users your way. In particular, I read your section on the differences between QuickCheck and Proptest and all of them seem like advantages in favor of proptest over quickcheck. Is this an exhaustive comparison? That is, does quickcheck have any advantages over proptest?

Apologies for the drive by comment. I haven't actually tried to use proptest yet, but figured I'd just get the ball rolling here. :)

[AltSysrq]

does quickcheck have any advantages over proptest?

A couple come to mind:

• Test case generation of complex data with quickcheck is much faster due to its stateless approach to shrinking.

• Quickcheck is more concise when using a lot of types with a single "canonical" generation strategy. There's some proposed work to bring similar support into proptest though.

• Quickcheck has support for more std types out-of-the-box.

To be honest, I've always had a "Am I holding it wrong?" sort of feeling with quickcheck (both in Rust and the one built in to fj) and there are definitely things like Cogen that I haven't been able to fully wrap my head around, so it's quite possible quickcheck has other advantages I don't see.

does this replace quickcheck completely?

I do think that for the majority of code-bases, moving from quickcheck to proptest wouldn't be too difficult (although it does require a bit of effort since a lot of edits have to be made); here's the one example I have of such a migration. The main exception I can think of is anything doing something interesting with custom shrinking, since custom shrinkers in proptest are more complex to implement.

Of course, my own opinion of how easy proptest is to use is rather skewed since I wrote it.

[BurntSushi]

@AltSysrq Awesome! Thanks for the reply. I'll add a link to QuickCheck's README to proptest and include a link to this issue.

[vitiral]

Can I just say that one of the things that makes the rust community so great is this right here -- library authors giving an honest discussion of the pros and cons of their own libraries, with the willingness (excitement even!) to point out alternatives to their own users.

This is what makes me so happy to work within the rust ecosystem.

❤️ @BurntSushi ❤️ @AltSysrq

[BurntSushi]

I opened a PR doing what I said I'd do a week ago. :-) BurntSushi/quickcheck#191

[Centril]

To @AltSysrq's third point, I am working on supporting more std types out-of-the-box with the goal of supporting all of them eventually (tho lifetimes in types makes difficult currently, but perhaps with ATCs we can move the possible on this.. - see #9).

[Centril]

Migration should be a bit easier now with proptest-quickcheck-interop - now you can essentially reuse quickcheck::Arbitrary implementations in proptest and use proptest combinators on that.

[AltSysrq]

Closing this since it looks like our discussion and all follow-ups have completed.

[leftaroundabout]

Dropping by as a Haskeller, I would (predictably) give a somewhat different view on the subject:

QuickCheck inherently expresses better what automated pseudo-exhaustive testing is all about.

The idea is that, in a strong static language, you want to write mostly total functions, i.e. functions that work for every possible value of the argument type. Thus, type-based testing is conceptually the right thing, and if you find yourself needing explicit generators with dedicated shrinking, it simply means your function takes its arguments in a too weak type. I.e. as it were, QuickCheck tests not only that your implementation of a given signature is right, it also checks that the signature is actually the right one!

Obviously this doesn't apply to Python because it doesn't use signatures in the first place, thus Hypothesis with its explicit generators is the right approach there. It also doesn't apply to C, where it's very common to write completely non-total functions (e.g. length parameter must match char pointer range). This is very different in Haskell where we take totalness seriously – mind, it isn't always achievable (that would require dependently-typed Agda), but in well over 90% of all cases it is^†.
About Rust I'm not sure, placed in between C++ and Haskell. IMO it does make sense to default to the conceptually more sound QuickCheck and only use Proptest in the (not uncommon) cases where it's necessary to stray far from totalness.

The unqualified comment by Hypothesis' author, “the way shrinking is handled in Haskell QuickCheck is bad” is at any rate nonsense.

---

^†There is one important exception: Haskell functions that expect a nonnegative number normally take it as an Int or Integer argument, though these include also negative values. This has partly historical, partly practical) reasons.

[Centril]

@leftaroundabout Hey fellow Haskeller =)

The unqualified comment by Hypothesis' author, “the way shrinking is handled in Haskell QuickCheck is bad” is at any rate nonsense.

I agree; Koen Claessen makes an interesting analysis here on the differences between QuickCheck and Hedgehog (which is more similar to Hypothesis): https://www.reddit.com/r/haskell/comments/646k3d/ann_hedgehog_property_testing/dg1485c/

However; I believe there's a lot of benefit to integrated shrinking in terms of convenience so that you don't have to encode shrinking separately. I believe a problem with the way the Arbitrary type class is defined in QC is that the minimum complete definition is only arbitrary and that shrink = [] can be left out. As a consequence, many Haskellers, especially aspiring ones simply do not define shrinking at all. When last I spoke with Koen, iirc he agreed that it was a mistake to include shrink = [] and not require an explicit definition always.

I believe that today, given that proptest also includes the trait Arbitrary, the library has practically moved much closer to QuickCheck and this shows that type based vs. integrated shrinking is not necessarily a useful distinction or opposites. It is entirely possible to define shrinking per type in proptest, tho probably not a good idea. If I'd classify proptest, I'd say that it really lies somewhere between Hedgehog and QuickCheck.

I also believe that mechanical deriving is easier when shrinking is part of the generator compared to if it is not.

Koen also writes that:

(1) Generating functions is not easily supported. In QuickCheck, we have the function promote :: (a -> Gen b) -> Gen (a->b) which you cannot implement for your type.

This is not the case for this crate; in a branch, I've implemented both CoArbitrary and defined Arbitrary for the moral equivalent of Fn(A) -> B. It might also be possible to also implement Fun a b and possibly partial concrete functions and integrate it in proptest.

There is however a key difference between QuickCheck the Haskell package and quickcheck the Rust crate. Namely, while Gen is a Monad in the former, this is not the case in the latter. I believe it is possible to define Gen as a monad tho - but it might have performance implications (which might not be important in a testing setting - or really important if you have a massive amount of properties and testing time is a problem for you).

[AltSysrq]

if you find yourself needing explicit generators with dedicated shrinking, it simply means your function takes its arguments in a too weak type

There are other cases where explicit generators are very useful. To use an example from my own experience, consider a function that takes a structure representing the contents of two filesystem directories and determines how to merge them without losing data. If you just throw literally random strings at it, you're very unlikely to hit a corner case where the two share a file with the same name; instead, you make a strategy to pick names from a much smaller set of strings. Also take for example the date parser in the proptest docs. Its domain is the set of all strings, but if you just pick 1000 strings from the set of all strings, the chance of making anything that even gets past its first couple "is the syntax valid" checks is vanishingly small; the test ends up only taking the trivial path.

the way shrinking is handled in Haskell QuickCheck is bad

While the article in question is perhaps unnecessarily strongly worded, I do agree with this sentiment, not on any theoretical grounds, but on practical: it's pretty difficult to write a complete and sensible shrinking function that operates solely on a value of the type. Some of this is just things like needing to manually convert the type to a tuple, shrink the tuple, and map it back to your type by hand for every type, which I assume is less of an issue in Haskell with its more powerful type system and functional support.

But this is also relevant for things like shrinking across enum variants. For example, maybe_ok(num::u64::ANY, ".*") (producingResult<u64, String>) can shrink the Ok case to an Err case, but you can't write a sensible function like

    fn shrink<T, E>(result: Result<T, E>) -> Result<T, E> {
        match result {
            Ok(v) if not_base_case(v) => /* ... */,
            Ok(_) => Err( /* ??? */),
            // snip
        }
    }

(I know this doesn't match the QC shrink signature, I'm just using concrete types and producing only one output value for simplicity's sake.)

[leftaroundabout]

FTR, I didn't mean to say that explicit generators don't make any sense. In particular, even if you have a properly total function taking, say, a list, it may have a high computional complexity and thus be infeasible to always take the (potentially quite long) lists thrown out by the standard instance. In this case, it can make a lot of sense to manually generate input. However, in this case you can still perfectly well use the standard shrinking function – usually, this will make the function cheaper, and even if it doesn't, it's not such a big deal when this only happens after a failed test.

About your examples I'm not convinced. For a parser, I'd say the primary way of testing should not start out with random strings at all, rather it should start out with arbitrary values of a Date type, use an (already tested) showing function to make a string of it, and test that the re-parsed result is the same as the original date you started with. Of course, starting with random strings (and small mutations of valid strings) is also important, but in that case the “first couple "is the syntax valid" checks” are precisely what you're interested in.

[AltSysrq]

For a parser …

Maybe we've been talking past each other, because you basically just described the date parser example in the tutorial (it just doesn't have a real Date type for the sake of keeping it small).

[leftaroundabout]

Actually, this is kind of my point: with QuickCheck, you arrive naturally at a test that gets to the heart of the matter, i.e. that tests values which are relevant for a date-parser instead of random strings. For this, QC forces you to define a type that expresses what values are relevant. In a real application, you'd probably want to have that type anyways.

With manual generators, you basically re-invent this type on the value / function level. Of course, that may sometimes be easier than actually defining the type as a type, especially in a language that doesn't offer great support for custom types. But, I'm somewhat reminded about Greenspun's Tenth Rule.