Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Example of property-based testing of the Vote structure #792

Closed
wants to merge 1 commit into from

Conversation

greg-szabo
Copy link
Member

This example shows how to test the Vote structure using quickcheck, the property-based testing tool.
Two properties of the Vote structure is tested:

  1. Protobuf identity: All Vote structs should be encodable to a protobuf bytestream and then the same Vote struct should be decoded from the stream.
  2. JSON identity: All Vote structs should be serializable as JSON structs and the same Vote struct should be deserialized from the string object.

You will see that three distinct errors are discovered by quickcheck:

  1. no timestamp error: The Timestamp property of the struct is public and optional. A Vote struct can be created with nil timestamp but we do not accept protobuf streams with no Timestamp.
  2. No such local time error: it is possible to create a Timestamp that is invalid (before the EPOCH).
  3. IntegerOverflow error: the generation of ValidatorIndex creates bigger numbers in some cases than allowed in ValidatorIndex. ValidatorIndex should only allow being created using valid numbers.

You can see that implementing these property tests requires all involved structs to have the Arbitrary trait implemented. This is yet another trait that we need and we only use it in testing. Possibly we could annotate them properly so they don't end up in the release binary.

  • Referenced an issue explaining the need for the change
  • Updated all relevant documentation in docs
  • Updated all code comments where relevant
  • Wrote tests
  • Updated CHANGELOG.md

@romac
Copy link
Member

romac commented Jan 27, 2021

Yeah having to incur a dependency on quickcheck and define these Arbitrary instances is quite annoying.

The proptest crate on the other hand allows you define your own strategy for generating arbitrary instances of a datatype independently of the datatype definition (for example in #709) and then use it within the test (for example in #709).

I like the proptest approach better since it decouples the strategy for generating test cases from the datatypes, but also allows such strategies to be composed together.

@shonfeder
Copy link
Contributor

I think we can close this in favor of #821, which builds on #815 using proptest?

@greg-szabo greg-szabo closed this Apr 9, 2021
@greg-szabo greg-szabo deleted the greg/pbt branch April 9, 2021 17:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants