Handle failures - to avoid long runs that break and then all compute time is lost

[DBCerigo]

Currently, the run loop runs all model-validation combinations and stores the result in memory and once all complete it then writes all the result to file. If a model-validation combination breaks, then whole loop breaks, and all the prior model-validation combination results are lost, resulting in wasted compute + waiting time for those model-validation combinations.

We could try to implement a solution to this.

[alex-hh]

Another option would be to incrementally add lines to csvs, rather than building all the results and then saving? This could also remove the pandas dependency if that is not used elsewhere.