QuickCheck-like fact support

[AlexBaranosky]

Maybe it could look something like:

(generated [a (midje/string) b (midje/string)]
  (fact
    (str a b) => (has-prefix a)))

Inspired by examples from ScalaCheck's site:
http://code.google.com/p/scalacheck/

[AlexBaranosky]

Midje could come prepackaged with some generators (like midje/string), and also supply a Generator protocol that could be used to supply a generator for any type of data a user has.

[AlexBaranosky]

It looks like ClojureCheck has already done a ton of the legwork on this.
https://bitbucket.org/kotarak/clojurecheck/src/85c314b1bd2b/src/main/clojure/clojurecheck/core.clj

I'm going to see if maybe we could utilize ClojureCheck as a library for this feature.

[AlexBaranosky]

This Agitator product for Java looks interesting. I wonder how it could be incorporated into Midje to improve the way TDD is done? http://www.agitar.com/pdf/Paper-Agitar-ODT-TDD.pdf [PDF]

[AlexBaranosky]

ClojureCheck:
http://kotka.de/blog/2010/06/ClojureCheck_is_back.html

I've contacted the maintainer of ClojureCheck about possibly pulling out the generator part of ClojureCheck into a separate namespace to use as a library in Midje.

Say you have a generative example like this, where any two strings concatenated together (str) always starts with the first string:

(generative [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

Besides the generative testing approach, I'm also interested in being able to run diagnostics on an expression. What if by swapping the first word (the macro) we could send feedback to standard out about useful tidbits of knowledge about the function under test? (Or maybe run the suite in diagnose mode or something)

(diagnose [?a (midje/string) ?b (midje/string)]
  (fact
     (str ?a ?b) => (has-prefix ?a)))

This is the feature of Agitator that seemed like it was thinking out of the box, and potentially really neat.

[AlexBaranosky]

Might make sense to use test.generative.generators as the library to use for Midje's (theoretical) generator library, since it seems better maintained:
https://github.com/clojure/test.generative/blob/master/src/main/clojure/clojure/test/generative/generators.clj

[AlexBaranosky]

"Yes, the code is doing what you want.
By the way, what else is it doing?"

[AlexBaranosky]

"It has been observed that TDD is an excellent methodology for developing “clean code that works” (Ron
Jeffries). Its many strengths include the ability to create code that does only what you want and to create
a thorough set of automated tests. However, by itself, TDD is incomplete as a coding and testing
methodology because of the fact that it creates code that does only what you want without taking into
account unintended side effects. ODT fills that void, using automated tools to test the behavior of code
and providing developers with actionable observations about possible unintended side effects."

[AlexBaranosky]

;; This first swipe at generative-style testing in Midje was super easy:

;; first a use of it

(defn make-string []
  (rand-nth ["a" "b" "c" "d" "e" "f" "g" "i"]))

(formula [a (make-string) b (make-string)]
  (str a b) => (has-prefix a))

;; How its defined

(def ^:dynamic *num-generations* 100)

(defmacro formula [bindings & body]
  (macro-for [_ (range *num-generations*)]
    `(let ~bindings
       (midje.sweet/fact
          ~@body))))

;; NOTES:
;; * just borrow some generator functions from test.generative, and voila
;; a stripped down generative test framework.
;; * this version makes a fact for every generation which would quickly
;; make the fact count in the report meaningless.
;; * doc-strings can be added later.

;; QUESTIONS:
;; * how can we make the reporting of these really nice?
;; * is this it? There has to be more to this kind of thing than simply
;; generating 100 of each formula. What do you think?

[AlexBaranosky]

I have looked into this further. What would it take to move this implementation from overly simplistic, to real-world usable?

1. store up the results of the 100 test runs, and then create only ONE report from them on failure
2. if there is a failure in one of the 100 runs, retry the fact execution with the inputs shrunken, repeat this process until it stops failing, then use the most shrunken failure case in the report output.

Both of these changes require adding interesting changes to Midje's flow of logic. Currently it assumes one fact per report, and that no fact would ever be re-ran.

[marick]

Re: "interesting changes to Midje's flow of logic". Can generative tests be considered tabular tests?

[AlexBaranosky]

Tabular tests will report a test run for each row of the table.

Each generative test would have 100 runs, so this will quickly report an insane # of test runs. Using tabular for generative testing wold also make it hard to do failure cases shrinking. (https://github.com/AlexBaranosky/Shrink) Because ideally it would be intelligent enough to stop when it sees a failure then go into a mode where it keeps calling shrink, until it no longer fails.

[marick]

How about using metaconstant notation instead of a logic variable notation? Consider: a metaconstant describes a value about which nothing is known except what's stated in a provided clause. A variable to be replaced by a value generator is one about which nothing is known except (in some vague sense) its type. So how about this test:

(fact
  (.startsWith ^String ..a.. ^String ..b..) ..a..) => true))

The existence of a tagged metaconstant gives Midje license to generate N tests.

[AlexBaranosky]

It's definitely an approach to consider. It is less boilerplate, which I like.

[AlexBaranosky]

@marick, I like your idea for a metaconstant syntax. But I think the logic variable notation is simpler to implement. My plan is to (attempt to) get it working using the logic syntax, then we can discuss further how to improve the syntax.

[AlexBaranosky]

Shrinking is harder than I erroneously imagined. To shrink we need to have a record of the inputs to the fact, but for normal facts (Which we piggy back over) there is no such record, as they are just baked into the fact code.

Ideally we need to be able to notice a failure in unprocessed.clj and then shrink the inputs and run another related fact.

I need to think on this.

[AlexBaranosky]

@marick:
I'd like to merge the formulas branch into master (except for the crufty last commit I accidentally pushed) [https://github.com/marick/Midje/commit/1bb5b25fbe30498000691ed1acb02a6ef47f8fe6]

It is alpha, and it doesn't shrink yet, but it is functional. I can mark the docstring ALPHA, but it seems worth getting it out in the open for people to try.

One question is d we want to suggest people use the generators built into test.generative (at least for now)? I personally have no problem with that, and think it might be the right final choice anyway (rather than recreating the wheel)...

[josephwilk]

Hows this coming along?
Curious to give it a try, is it documented in the wiki or anywhere else?

[marick]

The basics work now. Alex can say more. There's no documentation, but the tests can help:

https://github.com/marick/Midje/blob/master/test/behaviors/t_formulas.clj

[marick]

The original request exists: see formula. So I'll close this pending someone to take it on.