Generating functions

[DRMacIver]

One of the nice things Quickcheck can do is generate random functions. It would be nice if Hypothesis could do the same.

[szabba]

I've currently done one by hand while playing around

def integer_functions():
    return (
        integers().filter(
            lambda x: x != 0)
        .flatmap(
            lambda n: one_of(
                just(lambda x: x + n),
                just(lambda x: x - n),
                just(lambda x: x % n),
                just(lambda x: x * n),
                just(lambda x: x // n))))

How'd you envision the end result API when/if this gets implemented? I think something like

from hypothesis.strategies import functions, integers

@given(functions(int, str), integers()):
def test_generated_function_returns_a_string(f, x):
    assert isinstance(f(x), str)

would be nice.

Intuition tells me it'd require a library of building blocks, most of which would contain holes (places where other blocks can be dropped in). If one had type information about blocks and their holes (if any) it doesn't seem quite infeasible to generate those randomly... Am I missing something important from the bigger picture?

PS. Kudos for the library.

EDIT: I forgot about termination! In the presence of user-introduced blocks it might be infeasible to achieve it for all generated functions.

But maybe this can be tackled as a social issue - by providing a set of guidelines and best practices? Like Do not introduce blocks that do this-and-that and/or Throw an exception when your block recognises an input it can't possibly terminate for?

[Zac-HD]

If the functions can simply pretend to do something, I have a nice prototype of a composite strategy - it generates arbitrary lambdas that simply draw their return value from another strategy when called.

[Zac-HD]

Suggested API:

def functions(
	like: Callable[..., T]=None,
	returns: SearchStrategy[T]=None,
) -> SearchStrategy[Callable[..., T]]:

• Generated functions have an argspec matching like; defaulting to lambda *args, **kwargs: None.
• When called, the function draws from returns and returns that value. If None, infer from like or fall back to none().

[ib250]

Are we still looking at this? Coincidentally I sat down over one weekend thinking about this... I still think it'd be killer feature particularly in the scientific computing community

[ib250]

If we wanted to generate functions from the set theoretic perspective this would be equivalent to a lookup table with keys as the function inputs and outputs. The catch being we need to make the lookups total.

[DRMacIver]

Are we still looking at this?

"Looking" is a strong word. Certainly we'd like to have such a feature! I don't think any of us are currently actively planning to work on it, but it will probably happen at some point, and if you'd like to take a crack at it we'd be delighted to review a PR (feel free to open a WIP one if you want to sketch out a s solution).

I think the main difficulty is in pinning down the exact behaviour - what sort of functions do we want to return, how do we report their behaviour to the end user, etc.

[Zac-HD]

It's been around the middle of my todo list for a long time now, but obviously I haven't gotten to it yet - if you want to have a go that would be awesome!

For an initial version, I think it's more important to get the basic API right than to support everything. I've got some notes on that above, aiming to generate functions that can be used as callbacks. Specific notes:

• I would strongly prefer to make all generated "functions" nondeterministic, as we can add a pure=False argument later. Shrinking an example subject to the constraint that the call history is consistent is certainly not impossible, but will require lots of fiddly detail work with the internal APIs. Happy to take and help with a second PR, but let's land one without a lookup table first.
• hypothesis.internals.reflection.proxies is pretty much perfect for this, if the user passes a function to imitate. Implementation sketch:

@composite
def functions(draw, like=None, returns=None):
    """Docs here."""
    if like is None:
        like = lambda *args, **kwargs: None  # noqa
    elif not callable(like):
        raise InvalidArgument(...)
    if returns is None:
        returns = none()
    check_type(SearchStrategy, returns)

    @proxies(like)
    def inner(*args, **kwargs):
        val = draw(returns)
        note(
            "Called function: %s(%s) -> %r"
            % (like.__name__, arg_string(like, args, kwargs), val)
        )
        return val

    return inner

Note that the actual callback does not do any validation of it's arguments; in the general case this is extremely difficult and again I'd leave it for a future pull request.

[Zac-HD]

Somewhat more of a prototype: master...Zac-HD:functions-strat. Just needs testing, then a PR 😄