Why Spearmint stuck in bad hyperparam choices?

[cuihenggang]

Hello,

I really like the Spearmint tool. However, I find sometimes it makes weird behaviors that I could not understand (i.e., it keeps proposing hyperparam choices that are known to be bad, and never proposes hyperparam choices that are near the known minima).

Here is one example of such weird behavior. I have the config file like this (one float hyperparam and two integer hyperparams), and I was using spearmint-lite (I manually try each hyperparam that Spearmint proposes and report back the loss).
{
"X" : {
"name":"X",
"type":"float",
"min":-5,
"max":0,
"size":1
},
"Y" : {
"name":"Y",
"type":"int",
"min":0,
"max":3,
"size":1
},
"Z" : {
"name":"Y",
"type":"int",
"min":1,
"max":7,
"size":1
}
}

Here is the Spearmint result (since the time cost of running each hyperparam choice is the same, I just set the time cost to 1). I think it's quite obvious that the minima should be somewhere near {-2.5, 1, 4}, but I don't understand why Spearmint keeps letting me try {-5.0, 3, 7} and {-5.0, 3, 1}. Is it a bug of the implementation or Bayesian optimization is just not stable?

0 1 -5.0 0 1
-0.095405 1 -2.5 1 4
0 1 -1.25 0 6
0 1 -5.0 3 7
0 1 0.0 3 1
0 1 0.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 0 7
0 1 0.0 0 1
0 1 -2.3078 3 3
-0.0118625 1 -2.41399 3 7
0 1 -5.0 3 4
0 1 0.0 0 7
0 1 -2.53209 1 1
0 1 0.0 1 3
-0.0299104 1 -2.91007 1 7
0 1 -5.0 1 4
0 1 0.0 3 7
-0.062839 1 -3.23587 0 3
-0.00012286 1 -3.87776 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 0
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 0.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
0 1 -5.0 3 7
-6.6271e-05 1 -5.0 3 1
-6.6271e-05 1 -5.0 3 1
P P -5.0 3 1

[JasperSnoek]

Hi @cuihenggang, yes there are known pathologies with the optimization routine that have been addressed in a variety of follow up papers (e.g. https://arxiv.org/abs/1402.0929). Much of these improvements are implemented in the newer Spearmint codebase and its branches here (https://github.com/HIPS/Spearmint).

[cuihenggang]

Thank you @JasperSnoek for pointing this out. I think I should try the fixed version. I find in the HIPS/Spearmint repository, there are three branhces (master, PESC, and PESM). Which of the three branches do you suggest me to use?

Also, by the way, I really like the spearmint-lite interface that I don't need to install any database thing and can talk with Spearmint by just running the spearmint-lite.py command. With the new Spearmint codebase (HIPS/Spearmint), can I still do things like that?

Thank you very much!

[JasperSnoek]

Unfortunately, I don't think there's a "light" version of that code. It requires that you set up a database (mongodb).

I can't really speak for the various branches since I haven't been involved in some time. In terms of the academic literature PESM and PESC are more recent and should work better in theory. Maybe open an issue there?