Rapid is a Go library for property-based testing.
Rapid checks that properties you define hold for a large number of automatically generated test cases. If a failure is found, rapid automatically minimizes the failing test case before presenting it.
- Imperative Go API with type-safe data generation using generics
- Data generation biased to explore "small" values and edge cases more thoroughly
- Fully automatic minimization of failing test cases
- Persistence and automatic re-running of minimized failing test cases
- Support for state machine ("stateful" or "model-based") testing
- No dependencies outside the Go standard library
Here is what a trivial test using rapid looks like (playground):
package rapid_test
import (
"sort"
"testing"
"pgregory.net/rapid"
)
func TestSortStrings(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
s := rapid.SliceOf(rapid.String()).Draw(t, "s")
sort.Strings(s)
if !sort.StringsAreSorted(s) {
t.Fatalf("unsorted after sort: %v", s)
}
})
}More complete examples:
ParseDatefunction test: source code, playgroundQueuestate machine test: source code, playground
Rapid aims to bring to Go the power and convenience Hypothesis brings to Python.
Compared to testing.F.Fuzz, rapid shines
in generating complex structured data, including state machine tests, but lacks
coverage-guided feedback and mutations. Note that with
MakeFuzz, any rapid test
can be used as a fuzz target for the standard fuzzer.
Compared to gopter, rapid provides a much simpler API (queue test in rapid vs gopter), is much smarter about data generation and is able to minimize failing test cases fully automatically, without any user code.
As for testing/quick, it lacks both convenient data generation facilities and any form of test case minimization, which are two main things to look for in a property-based testing library.
Suppose we've written arithmetic functions add, subtract and multiply
and want to test them. Traditional testing approach is example-based —
we come up with example inputs and outputs, and verify that the system behavior
matches the examples:
func TestArithmetic_Example(t *testing.T) {
t.Run("add", func(t *testing.T) {
examples := [][3]int{
{0, 0, 0},
{0, 1, 1},
{2, 2, 4},
// ...
}
for _, e := range examples {
if add(e[0], e[1]) != e[2] {
t.Fatalf("add(%v, %v) != %v", e[0], e[1], e[2])
}
}
})
t.Run("subtract", func(t *testing.T) { /* ... */ })
t.Run("multiply", func(t *testing.T) { /* ... */ })
}In comparison, with property-based testing we define higher-level properties that should hold for arbitrary input. Each time we run a property-based test, properties are checked on a new set of pseudo-random data:
func TestArithmetic_Property(t *testing.T) {
rapid.Check(t, func(t *rapid.T) {
var (
a = rapid.Int().Draw(t, "a")
b = rapid.Int().Draw(t, "b")
c = rapid.Int().Draw(t, "c")
)
if add(a, 0) != a {
t.Fatalf("add() does not have 0 as identity")
}
if add(a, b) != add(b, a) {
t.Fatalf("add() is not commutative")
}
if add(a, add(b, c)) != add(add(a, b), c) {
t.Fatalf("add() is not associative")
}
if multiply(a, add(b, c)) != add(multiply(a, b), multiply(a, c)) {
t.Fatalf("multiply() is not distributive over add()")
}
// ...
})
}Property-based tests are more powerful and concise than example-based ones — and are also much more fun to write. As an additional benefit, coming up with general properties of the system often improves the design of the system itself.
As you've seen from the examples above, it depends on the system you are testing. Usually a good place to start is to put yourself in the shoes of your user and ask what are the properties the user will rely on (often unknowingly or implicitly) when building on top of your system. That said, here are some broadly applicable and often encountered properties to keep in mind:
- function does not panic on valid input data
- behavior of two algorithms or data structures is identical
- all variants of the
decode(encode(x)) == xroundtrip
At its core, rapid does a fairly simple thing: generates pseudo-random data based on the specification you provide, and check properties that you define on the generated data.
Checking is easy: you simply write if statements and call something like
t.Fatalf when things look wrong.
Generating is a bit more involved. When you construct a Generator, nothing
happens: Generator is just a specification of how to Draw the data you
want. When you call Draw, rapid will take some bytes from its internal
random bitstream, use them to construct the value based on the Generator
specification, and track how the random bytes used correspond to the value
(and its subparts). This knowledge about the structure of the values being
generated, as well as their relationship with the parts of the bitstream
allows rapid to intelligently and automatically minify any failure found.
Property-based testing focuses on quick feedback loop: checking the properties on a small but diverse set of pseudo-random inputs in a fractions of a second.
In comparison, fuzzing focuses on slow, often multi-day, brute force input generation that maximizes the coverage.
Both approaches are useful. Property-based tests are used alongside regular
example-based tests during development, and fuzzing is used to search for edge
cases and security vulnerabilities. With
MakeFuzz, any rapid test
can be used as a fuzz target.
Just run go test as usual, it will pick up also all rapid tests.
There are a number of optional flags to influence rapid behavior, run
go test -args -h and look at the flags with the -rapid. prefix. You can
then pass such flags as usual. For example:
go test -rapid.checks=10_000Rapid is stable: tests using rapid should continue to work with all future rapid releases with the same major version. Possible exceptions to this rule are API changes that replace the concrete type of parameter with an interface type, or other similar mostly non-breaking changes.
Rapid is licensed under the Mozilla Public License Version 2.0.