Compare sampled normal distribution to PDF #1121
Conversation
Those will only be used for testing distributions.
`Normal` is sampled many times into a histogram, which is then compared to the expected probability density function. To make debugging easier, sparklines of the expected distribution, the histogram, and their difference are printed. Currently, the test fails if the difference is significantly larger than the expected error of the histogram bin. However, the error estimate does not take the error in the normalization due to the finite width of the histogram into account. This should not be a problem, as long as the distribution is almost zero outside the range covered by the histogram. In principle, this approach can be generalized to other distributions.
|
Excellent! One million samples: what is the run-time, and is this accurate enough to be really useful? Should we run it in all test-suite runs or put it behind a feature flag and limit when it runs? |
|
We could verify other qualities as well. Mean, standard deviation, skewness, kurtosis, etc. See https://github.com/miloyip/normaldist-benchmark#procedure |
One million samples was quite accurate already for the chosen bins (as shown above, the absolute maximum error per bin was ca. 0.006 in probability units). I should probably calculate the relative errors though to make sure. 100 000 samples also gives you a perfect sparkline, but for 10 000 samples you get visible differences.
On my CPU the runtime was unnoticeable. I don't think it will be a problem, but let me check the runtime on the CI after I fixed the tests.
We could, and this is already supported by |
|
The new tests run in about half a second, which I think is acceptable even if extended to all distributions. |
|
@dhardy I think I addressed your comments? |
This tests that we are sampling the normal distribution correctly. For now, this is limited to the normal distribution, but it can be extended to other distributions as well. I'm using sparklines to make debugging easier:
(This looks a bit better in the terminal.)
Refs #357.