Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate the configuration produced by LlamaTune inverse_transform() #806

Closed
wants to merge 12 commits into from
Closed

Validate the configuration produced by LlamaTune inverse_transform() #806

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4b55dbb
Validate the configuration produced by LlamaTune inverse_transform()
motus Jul 17, 2024
57fde9c
Merge branch 'main' into sergiym/opt/llamatune_inv
motus Jul 22, 2024
b58236f
Update azure credentials to be more flexible (#787)
eujing Jul 19, 2024
8034458
Fix mypy 1.11 warnings (#809)
motus Jul 22, 2024
ca898ed
Fix the coercion of scores to floats in the optimizer (#789)
motus Jul 22, 2024
f57d480
Merge remote-tracking branch 'sergiy/sergiym/opt/llamatune_inv' into …
bpkroth Jul 22, 2024
13fb61f
comments
bpkroth Jul 22, 2024
f02411a
comments
bpkroth Jul 22, 2024
b4d5007
apply the q_scaler
bpkroth Jul 22, 2024
e0e3b30
more tweaks
bpkroth Jul 22, 2024
1a44544
more fixups
bpkroth Jul 22, 2024
ceebf90
reorg to make pylint happier
bpkroth Jul 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions mlos_core/mlos_core/spaces/adapters/llamatune.py
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Somewhat unrelated, we should add "quantization + llamatune" tests as well.

Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,8 @@ def inverse_transform(self, configurations: pd.DataFrame) -> pd.DataFrame:
self.target_parameter_space,
vector=target_config_vector,
)
self.target_parameter_space.check_configuration(target_config)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might just need to clip them to the range in the case that it's too large (e.g., due to quantization issues).
Are there other failure cases than that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a slightly closer look at this.
max_unique_values_per_param is forcing the lower dimensions space to be UniformIntegerHyperparatmers, but the vector produced that is then dot producted with the approximate inverted matrix returns floats.

So, I think there's an additional scaling operation required to put that vector back into the max_unique_values_per_param range when that option is not None.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I've added the q_scaler inverse transform here as well, which makes the values look much better, but one is still out of bounds. I think it's maybe just needing clipping either there, or in a previous step.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kkanellis

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

pytest -n0 mlos_bench/mlos_bench/tests/optimizers/llamatune_opt_test.py

to reproduce

# target_config.check_valid_configuration() # for ConfigSpace 1.0

target_configurations.append(target_config)

Expand Down
Loading