a bit cleaner definition of test config spaces #60
Conversation
| @@ -73,7 +73,7 @@ def test_mock_optimization_loop(mock_env_no_noise: MockEnv, mock_opt: MockOptimi | |||
| "vmSize": "Standard_B2ms", | |||
| "idle": "halt", | |||
| "kernel_sched_migration_cost_ns": 117026, | |||
| "kernel_sched_latency_ns": 149827700, | |||
| "kernel_sched_latency_ns": 149827706, | |||
There was a problem hiding this comment.
These really should be getting returned on the quantization boundary, no?
There was a problem hiding this comment.
The proper quantization boundary here should be something like 100000000..
The problem is that neither Mock Optimizer nor FLAML respect quantization (or distributions, for that matter). MockOptimizer just samples uniformly from the Tunables' range and does not use ConfigSpace at all; For FLAML we have a very limited conversion from ConfigSpace to FLAML hyperparameters that just ignores all that info (and I don't know if FLAML has any support for quantization etc. internally)
I still have some faint hope for SMAC3, but I cannot figure out why it does not call ConfigSpace.sample_configuration() like I think it should - maybe it also taps directly into the hyperparameters' ranges etc. and uses ConfigSpace only to validate the proposed configuration
There was a problem hiding this comment.
Yeah, I see now. Might need to implement quantization in the mlos_bench Tunables layer, but that might interact poorly with the underlying optimizer (e.g., register a quantized version of the actual values suggested).
…h/MLOS into quantization-hack-cleanup
No description provided.