QCheck

QuickCheck inspired property-based testing for OCaml, and combinators to generate random values to run tests on.

The documentation can be found here. This library spent some time in qtest, but is now standalone again!

To construct advanced random generators, the following libraries might be of interest:

• Feat

• @gasche’s generator library

Jan Midtgaard (@jmid) has a lecture about property-based testing that relies on QCheck.

Table of Contents

• Use
• Build
• License
• An Introduction to the Library
  • List Reverse is Involutive
  • Mirrors and Trees
  • Preconditions
  • Long tests
  • Runners
  • Integration within OUnit
  • Integration within alcotest
  • Integration within Rely
  • Deriver
  • Compatibility notes

Use

See the documentation. I also wrote a blog post that explains how to use it and some design choices; however, be warned that the API changed in lots of small ways (in the right direction, I hope) so the code will not work any more. An Introduction to the Library is an updated version of the blog post’s examples.

Build

$ make

You can use opam:

$ opam install qcheck

License

The code is now released under the BSD license.

An Introduction to the Library

First, let’s see a few tests. Let’s open a toplevel (e.g. utop) and type the following to load QCheck:

#require "qcheck";;

Note	alternatively, it is now possible to locally do: dune utop src to load qcheck.

List Reverse is Involutive

We write a random test for checking that List.rev (List.rev l) = l for any list l:

let test =
  QCheck.Test.make ~count:1000 ~name:"list_rev_is_involutive"
   QCheck.(list small_nat)
   (fun l -> List.rev (List.rev l) = l);;

(* we can check right now the property... *)
QCheck.Test.check_exn test;;

In the above example, we applied the combinator list to the random generator small_nat (ints between 0 and 100), to create a new generator of lists of random integers. These builtin generators come with printers and shrinkers which are handy for outputting and minimizing a counterexample when a test fails.

Consider the buggy property List.rev l = l:

let test =
  QCheck.Test.make ~count:1000 ~name:"my_buggy_test"
   QCheck.(list small_nat)
   (fun l -> List.rev l = l);;

When we run this test we are presented with a counterexample:

# QCheck.Test.check_exn test;;
Exception:
QCheck.Test.Test_fail ("my_buggy_test", ["[0; 1] (after 23 shrink steps)"]).

In this case QCheck found the minimal counterexample [0;1] to the property List.rev l = l and it spent 23 steps shrinking it.

Now, let’s run the buggy test with a decent runner that will print the results nicely (the exact output will change at each run, because of the random seed):

# QCheck_runner.run_tests [test];;

--- Failure --------------------------------------------------------------------

Test my_buggy_test failed (10 shrink steps):

[0; 1]
================================================================================
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

For an even nicer output QCheck_runner.run_tests also accepts an optional parameter ~verbose:true.

Mirrors and Trees

QCheck provides many useful combinators to write generators, especially for recursive types, algebraic types, and tuples.

Let’s see how to generate random trees:

type tree = Leaf of int | Node of tree * tree

let leaf x = Leaf x
let node x y = Node (x,y)

let tree_gen = QCheck.Gen.(sized @@ fix
  (fun self n -> match n with
    | 0 -> map leaf nat
    | n ->
      frequency
        [1, map leaf nat;
         2, map2 node (self (n/2)) (self (n/2))]
    ));;

(* generate a few trees, just to check what they look like: *)
QCheck.Gen.generate ~n:20 tree_gen;;

let arbitrary_tree =
  let open QCheck.Iter in
  let rec print_tree = function
    | Leaf i -> "Leaf " ^ (string_of_int i)
    | Node (a,b) -> "Node (" ^ (print_tree a) ^ "," ^ (print_tree b) ^ ")"
  in
  let rec shrink_tree = function
    | Leaf i -> QCheck.Shrink.int i >|= leaf
    | Node (a,b) ->
      of_list [a;b]
      <+>
      (shrink_tree a >|= fun a' -> node a' b)
      <+>
      (shrink_tree b >|= fun b' -> node a b')
  in
  QCheck.make tree_gen ~print:print_tree ~shrink:shrink_tree;;

Here we write a generator of random trees, tree_gen, using the fix combinator. fix is sized (it is a function from int to a random generator; in particular for size 0 it returns only leaves). The sized combinator first generates a random size, and then applies its argument to this size.

Other combinators include monadic abstraction, lifting functions, generation of lists, arrays, and a choice function.

Then, we define arbitrary_tree, a tree QCheck.arbitrary value, which contains everything needed for testing on trees:

• a random generator (mandatory), weighted with frequency to increase the chance of generating deep trees

• a printer (optional), very useful for printing counterexamples

• a shrinker (optional), very useful for trying to reduce big counterexamples to small counterexamples that are usually more easy to understand.

The above shrinker strategy is to

• reduce the integer leaves, and

• substitute an internal Node with either of its subtrees or by splicing in a recursively shrunk subtree.

A range of combinators in QCheck.Shrink and QCheck.Iter are available for building shrinking functions.

We can write a failing test using this generator to see the printer and shrinker in action:

let rec mirror_tree (t:tree) : tree = match t with
  | Leaf _ -> t
  | Node (a,b) -> node (mirror_tree b) (mirror_tree a);;

let test_buggy =
  QCheck.Test.make ~name:"buggy_mirror" ~count:200
    arbitrary_tree (fun t -> t = mirror_tree t);;

QCheck_runner.run_tests [test_buggy];;

This test fails with:

--- Failure --------------------------------------------------------------------

Test mirror_buggy failed (6 shrink steps):

Node (Leaf 0,Leaf 1)
================================================================================
failure (1 tests failed, 0 tests errored, ran 1 tests)
- : int = 1

With the (new found) understanding that mirroring a tree changes its structure, we can formulate another property that involves sequentializing its elements in a traversal:

let tree_infix (t:tree): int list =
  let rec aux acc t = match t with
    | Leaf i -> i :: acc
    | Node (a,b) ->
      aux (aux acc b) a
  in
  aux [] t;;

let test_mirror =
  QCheck.Test.make ~name:"mirror_tree" ~count:200
    arbitrary_tree
    (fun t -> List.rev (tree_infix t) = tree_infix (mirror_tree t));;

QCheck_runner.run_tests [test_mirror];;

Preconditions

The functions QCheck.assume and QCheck.(=⇒) can be used for tests with preconditions. For instance, List.hd l :: List.tl l = l only holds for non-empty lists. Without the precondition, the property is false and will even raise an exception in some cases.

let test_hd_tl =
  QCheck.(Test.make
    (list int) (fun l ->
      assume (l <> []);
      l = List.hd l :: List.tl l));;

QCheck_runner.run_tests [test_hd_tl];;

Long tests

It is often useful to have two version of a testsuite: a short one that runs reasonably fast (so that it is effectively run each time a projet is built), and a long one that might be more exhaustive (but whose running time makes it impossible to run at each build). To that end, each test has a 'long' version. In the long version of a test, the number of tests to run is multiplied by the ~long_factor argument of QCheck.Test.make.

Runners

The module QCheck_runner defines several functions to run tests, including compatibility with OUnit. The easiest one is probably run_tests, but if you write your tests in a separate executable you can also use run_tests_main which parses command line arguments and exits with 0 in case of success, or an error number otherwise.

Integration within OUnit

OUnit is a popular unit-testing framework for OCaml. QCheck provides a sub-library qcheck-ounit with some helpers, in QCheck_ounit, to convert its random tests into OUnit tests that can be part of a wider test-suite.

let passing =
  QCheck.Test.make ~count:1000
    ~name:"list_rev_is_involutive"
    QCheck.(list small_nat)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    ~name:"fail_sort_id"
    QCheck.(list small_nat)
    (fun l -> l = List.sort compare l);;

let _ =
  let open OUnit in
  run_test_tt_main
    ("tests" >:::
       List.map QCheck_ounit.to_ounit_test [passing; failing])

Note	the package qcheck contains the module QCheck_runner which contains both custom runners and OUnit-based runners.

Integration within alcotest

Alcotest is a simple and colorful test framework for OCaml. QCheck now provides a sub-library qcheck-alcotest to easily integrate into an alcotest test suite:

let passing =
  QCheck.Test.make ~count:1000
    ~name:"list_rev_is_involutive"
    QCheck.(list small_int)
    (fun l -> List.rev (List.rev l) = l);;

let failing =
  QCheck.Test.make ~count:10
    ~name:"fail_sort_id"
    QCheck.(list small_int)
    (fun l -> l = List.sort compare l);;


let () =
  let suite =
    List.map QCheck_alcotest.to_alcotest
      [ passing; failing]
  in
  Alcotest.run "my test" [
    "suite", suite
  ]

Integration within Rely

Rely is a Jest-inspire native reason testing framework. @reason-native/qcheck-rely is available via NPM and provides matchers for the easy use of qCheck within Rely.

open TestFramework;
open QCheckRely;

let {describe} = extendDescribe(QCheckRely.Matchers.matchers);

describe("qcheck-rely", ({test}) => {
  test("passing test", ({expect}) => {
    let passing =
      QCheck.Test.make(
        ~count=1000,
        ~name="list_rev_is_involutive",
        QCheck.(list(small_int)),
        l =>
        List.rev(List.rev(l)) == l
      );
    expect.ext.qCheckTest(passing);
    ();
  });
  test("failing test", ({expect}) => {
    let failing =
      QCheck.Test.make(
        ~count=10, ~name="fail_sort_id", QCheck.(list(small_int)), l =>
        l == List.sort(compare, l)
      );

    expect.ext.qCheckTest(failing);
    ();
  });
});

Deriver

A ppx_deriver is provided to derive QCheck generators from a type declaration.

type tree = Leaf of int | Node of tree * tree
[@@deriving qcheck]

See the according README for more information and examples.

Compatibility notes

Starting with 0.9, the library is split into several components:

• qcheck-core depends only on unix and bytes. It contains the module QCheck and a QCheck_base_runner module with our custom runners.

• qcheck-ounit provides an integration layer for OUnit

• qcheck provides a compatibility API with older versions of qcheck, using both qcheck-core and qcheck-ounit. It provides QCheck_runner which is similar to older versions and contains both custom and Ounit-based runners.

• qcheck-alcotest provides an integration layer with alcotest

Normally, for contributors, opam pin https://github.com/c-cube/qcheck will pin all these packages.