New performance threshold for nightly tests

[djdameln]

Currently, the nightly tests use a model- and category-specific performance score threshold, obtained by running a number of repeated training runs, and collecting the lowest observed scores per model and category. This is problematic, because it's always possible to observe an even lower score without anything being wrong with the model.

I propose the following alternative solution:

1. Run a number of repetitions of each model and category combination, and collect the observed performance score.
2. Estimate the probability distribution of the observed scores, for each model and category combination (I assume the observed scores are normally distributed).
3. Compute the lower bound of the 95% (one-tailed) confidence interval, and use this as the failure threshold in the tests.
4. For any failed test, we now know that there is a 5% chance of false positive. We can use the flaky package to allow multiple attempts.

[ashwinvaidya17]

Another idea (or in addition) can be to run benchmarking script in the nightly and collect the csv file. We can then track these metrics each day to see if there is any constant decrease in performance for a model.