Stopping criterion in Spearmint

[abhishekmalali]

Hi,

I am trying to use spearmint for parameter optimization. Like any optimization problem, there can be many solutions. What stopping criterion do you use? I can imagine it being something like tolerance as you get closer to the minima/maxima . I wanted to know if there is a way I can stop it manually. Just in case I fell that the model might not converge!

[Nitti121]

Hi,
I am also facing the issue regarding stopping criterion. Have you run branin example?..Is it showing the final optimized results??

[JasperSnoek]

The short answer is that there isn't a stopping criterion, but rather you can just stop the optimization when it looks like the objective isn't getting any better (or your patience runs out).

The long answer is that it is difficult to assess something like a notion of convergence in the global optimization of non-convex and noisy functions. It is essentially impossible to ever know for certain whether you have found the global optimum, or whether you're close to it (e.g. it's easy to construct functions with extremely deep and narrow modes that are hard to find). So then a stopping criterion becomes a question of when have you found a solution that is "good enough", which is more of a problem specific and subjective question. Rather than elicit criteria from the user and possibly implement a cascade of heuristics, we decided that ctrl-c was probably a better stopping criterion for now.

[abhishekmalali]

@JasperSnoek - Thank you for your answer! I have one more question. Is there a way to retrieve the results after using ctrl-c as a stopping criterion? Because all processing is linked to MongoDB which I am again not sure was integrated.

I have a suggestion. How about incorporating a keyword argument for early stopping where we can fix the number of models to be evaluated. It will be similar to random search but will be much smarter. This is assuming the optimization surface is not skewed.