Derive generators from type definitions

[evnu]

This merge request adds a macro to derive generators automatically from type
definitions @type. Certain restrictions apply, such as not being able
to generate a pid or reference. The macro can be used to create
generators for all public types within a module:

use PropCheck.DeriveGenerators

It can also be used to create generators for other modules, as long as
they are already compiled:

use PropCheck.DeriveGenerators, module: OtherModule

The macro creates a new module module.Generate, which contains the
generators. For each @type t, a function Generate.t() is created,
which can then be used as a generator in forall. For types
@type t(var) taking a type variable, the function expects a generator
as an argument for each type variable.

To allow adding types which are cannot be turned into a generator, the
option :only can be used.

use PropCheck.DeriveGenerators, only: [t: 0]

With this, only the type t will be turned into a generator.

Fix #69

[evnu]

@alfert I present this PR in an early stage to get some feedback. Most of the needed functionality is implemented (see the unit test for a comprehensive example), but besides the documentation, I think this needs a second set of eyes. Note that I did not implement deriving test functions from @spec, as this does not seem too useful to me right now. But with the given expansion functions, this should also be doable, as long as the AST is similar to the AST of @type definitions.

[evnu]

I forgot to mention, I did not yet think about recursive types (#4, #5). For now, I would leave that out and instead throw an NotSupported error. But I would be willing to put some time in this, if we find some nice examples and what a good generator would look like.

[evnu]

Specifically, this pr currently targets #3.

[alfert]

I did not yet think about recursive types

For non-recursive types, this is rather simple (as we both showed by example), it is only lengthy and tedious code.

Generatring recursive types are where the meat is. They are complicated to code and it is questionable whether this is an effective approach for testing. Typically, an automatically derived generator can only be the start of the test engineering process, see the lengthy stack example in the tests: you need to define a random distribution between the various recursive branches to generate all variants. This distribution decide massively on the shape and size of the generated data structures. In my experience, there is no one-size-fits-all generator for complicated stuff. Thus, I only started the adventure of generating recursive generators, but never came to an end.

From the perspective: are non-recursive type still a useful feature? Or are there other development usability or feature enhancements for PropCheck we should follow instead (or additionally)?

[evnu]

From the perspective: are non-recursive type still a useful feature?

In my experience, most types are non-recursive. At least most custom types I usually use in my code are non-recursive. Hence I would argue that those simple cases can indeed be a nice usability feature (see for example https://github.com/alfert/propcheck/pull/128/files#diff-50c824f0c4ab83bd425c63d90fe2aa7bR10 in this PR; it feels kind-of right to be able to get a simple generator almost for free).

Of course you are right w.r.t. recursive types, they are complicated. I wonder if it would still be nice to have automatic generators which work "most of the time" and clearly communicate the restrictions when it comes to custom recursive types.

Note that I wouldn't be angry if this PR here does not lead to a merged commit on master. I found it already worthwhile to play around and get a feeling were the actual problems with this approach are :)

[evnu]

I liked this comment by Kostis (mailing list):

The PropEr way to deal with recursive types is to delay writing your own generators as long as possible. You can start with writing just the recursive type definitions and use them as a "for free" generator.

This captures what I mean above. Maybe "good enough" is what is needed most of the time.

[alfert]

In my experience, most types are non-recursive.

This is true indeed, if you are writing application code with a (relational) database in mind. All functional data structures are usually recursive, but with lists, structs and maps at hand, it could be possible to avoid recursive data structure in most practical places.

If we follow you approach, then there is the obligation to document it properly, since it contradicts my statements in the README. Also the limits of the approach needs to be documented clearly.

The technical problem of recursive types is that they may be mutually recursive in an indirect manner while spawning across several modules at the same time. You need to detect and parse these structures properly and you need to store them somehow while creating the generators. That is what the type server in PropEr is for, but this nice beast is not intended to be used from Elixir or from outside of the internals from PropEr. If I remember right, this was the unpleasant experience that drove me to abandon the automatic generators.

[evnu]

This is true indeed, if you are writing application code with a (relational) database in mind. All functional data structures are usually recursive, but with lists, structs and maps at hand, it could be possible to avoid recursive data structure in most practical places.

So, at the very least, I would like to detect problematic recursive types with cycle detection and throw an error. Otherwise, users would probably run into problems which are very hard to debug.

The technical problem of recursive types is that they may be mutually recursive in an indirect manner while spawning across several modules at the same time.

Yes. You are right that this needs some kind of type server. I don't have an intuitive understanding yet how that would look like. The :digraph module seems like a good first fit, but I found its API lacking. For example, it is not simple to output a formatted forest of types without explicitly knowing which types are root nodes in the tree. I will play around more with that.

All in all, this thing is a very interesting problem indeed. I got a better understanding now where some of the complexity in PropEr comes from :)

[alfert]

There is the libgraph library on Hex which might be an interesting alternative to :digraph.

[evnu]

There is the libgraph library on Hex which might be an interesting alternative to :digraph.

Thanks, I will take a look at it.

[evnu]

I spent some time since November in implementing this. After some time, I decided to decouple this from PropCheck to simplify playing around with it. The result is an application :propcheck_derive.

I made a lot of progress and extended the simple derive generators with a type server, which checks if recursive types are present. The implementation can be found here. I haven't released this yet on hex.pm, as there is still some fleshing out to do. Nevertheless, I was able to create generators for all types in Elixir's standard lib, where the types do not have cycles and no non-implemented types such as pid() are used. See here for the list of Elixir modules which did not make the implementation crash.