Use number of bins instead of quantization interval in mlos_bench tunables #835

motus · 2024-08-06T22:09:39Z

Closes #803

Future PR will rename the config schema in order to reduce confusion on the change in semantics, but also keep this PR smaller.

…ntization

…nch (configs and internally)

…y value of tunable)

…t tests

bpkroth · 2024-08-13T18:26:36Z

@bpkroth before you merge, can you please run tests locally for this PR? One unit test, toy_optimization_loop_test.py::test_flaml_optimization_loop, now produces results on GitHub that are different from what we've had before and what I am getting on my laptop. I have not had time to investigate the issue yet.

This is due to a bump in FLAML:

mlos_bench/mlos_bench/optimizers/convert_configspace.py

mlos_bench/mlos_bench/tunables/tunable.py

mlos_bench/mlos_bench/tests/optimizers/toy_optimization_loop_test.py

Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

...chemas/tunable-params/test-cases/bad/invalid/tunable-params-int-bad-float-quantization.jsonc

…-cases/bad/invalid/tunable-params-int-bad-float-quantization.jsonc Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>

mlos_bench/mlos_bench/tests/tunables/tunable_to_configspace_test.py

mlos_bench/mlos_bench/tunables/tunable.py

mlos_bench/mlos_bench/optimizers/convert_configspace.py

mlos_bench/mlos_bench/tests/tunables/test_tunables_size_props.py

unit tests to produce nice human-readable values

bpkroth

LGTM, let's push the change in terminology (quantized_num_bins or whatever we decided) soon though.

Merge after (or instead of) #835 diff from #835 :: https://github.com/motus/MLOS/pull/15/files Closes #803 --------- Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com> Co-authored-by: Brian Kroth <bpkroth@microsoft.com>

motus added 4 commits August 5, 2024 16:38

monkey patch ConfigSpace.Hyperparameter.sample_value() with value qua…

7f536a8

…ntization

add quantization to some tunables in unit tests

f54587e

Merge branch 'main' into sergiym/tunable/quantization

d4516db

use number of bins instead of quantization interval length in mlos_be…

86e1557

…nch (configs and internally)

motus added WIP Work in progress - do not merge yet mlos-bench labels Aug 6, 2024

motus self-assigned this Aug 6, 2024

motus added 16 commits August 6, 2024 15:40

fix quantized_values to use quantization bins

61e9cc2

typo: accidentally removed quantization defs

de657a4

make tunable size tests work

ecc4b75

minor fix to the grid search unit tests (new data type for cardinalit…

5118454

…y value of tunable)

mypy fixes to grid search

ef0486d

disable pylint warning in _tunable_to_configspace

565b3d7

minor docstring formatting

1c8ca09

Merge branch 'main' into sergiym/tunable/quantization_bins

5087c94

move monkey patching to a standalone function that can be used in uni…

28731c6

…t tests

better type annotations

2b6c863

add quantization to ConfigurationSpace parameters in unit tests

6c5401b

docstring formatting

ebe8730

better parameter naming

6b76039

docstring fix

3e984e1

add quantization patching unit tests; bugfix in monkey patch code

1b13e2d

imports formatting

d6eace8

motus mentioned this pull request Aug 8, 2024

Monkey patch Hyperparameter.sample_value() to support quantization #833

Closed

bpkroth mentioned this pull request Aug 8, 2024

Add llamatune checks with quantized values #814

Merged

motus added 5 commits August 8, 2024 19:21

add unit tests for repatching the same method multiple times

eaddd89

fix the repatching unit test

7c8efac

fix the repatching unit test

7e8c6b4

try a new value of optim. score for FLAML

cff2eb7

new version of FLAML produces a different config

53befe4

motus added ready for review Ready for review and removed WIP Work in progress - do not merge yet labels Aug 9, 2024