Use Hypothesis for testing

[brandonwillard]

Given the clear constraints of our problem domain(s), the amount of repetition found within our test suite, the lack of viable test automations, etc., it might be worth using Hypothesis. That seems like a good platform upon which we can build more extensible and robust testing.

[bergesalexis]

Hi ! I'm interested in helping out for this. @AlexAndorra is a friend and talked to me about this issue.

I don't have any background in OSS, but have been coding in python for the past 7 years, and I am at the head of a team of 10 data scientist + data engineer in my current job, so I am quite familiar with the importance of test suites, and of proper CI/CD :). I am also already familiar with Hypothesis (we are just starting to use it in my team), I have forked Theano-PyMC and run all the tests locally.

The test suite and code base are already large though, if you had any pointers for me, this would be very much appreciated. Ideally this would be a very small thing to begin with in order to experience your coding guidelines + PR + code review process end to end and be more efficient after that. Many thanks in advance.

[brandonwillard]

#141 and/or #16 are probably the best places to start. They're both rather high-level changes, so you would need to get a broad overview of the elements involved before making changes. Unfortunately, there's no avoiding that.

#16 has more specifics to work with, but #141 is the simplest in terms of the general idea and perhaps the most immediately measurable improvement that could be made between the two.

Neither of them are likely to be "fixed" by the use of Hypothesis, though. Instead, they would serve as a good means of understanding the general situation and perhaps finding some good explicit use cases for Hypothesis.

Otherwise, the tests that seem more immediately appropriate for Hypothesis are probably the ones in tests.tensor.test_basic and tests.tensor.test_basic_scipy. Those tests construct and reuse a lot of value ranges for functions.

If you want to start looking into any of those, you can follow up with questions in the respective issues.

[majidaldo]

Why specifically Hypothesis when you can have parameterized data generators?

[etellman]

It looks like this is a pretty old issue that hasn't seen much activity lately, but I was interested in it, so I did a quick POC for using Hypothesis.

See #1507

I'm an Aesara novice, so I picked a test that seemed really simple and easy to convert as an example.

[etellman]

Why specifically Hypothesis when you can have parameterized data generators?

It's a different way of thinking about tests. You specify properties that should always be true and Hypothesis tries to find inputs that make the property false. If it finds something that works, it then tries to narrow down the value to the one that would be most helpful for identifying the problem. For instance, if large values seem to be problematic, Hypothesis tries to find the smallest value that's big enough to make the test fail.