Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mlos_bench config json schema validation - storage #349

Merged
merged 122 commits into from
May 16, 2023
Merged

mlos_bench config json schema validation - storage #349

merged 122 commits into from
May 16, 2023

Conversation

bpkroth
Copy link
Contributor

@bpkroth bpkroth commented May 10, 2023 •
edited
Loading

Extends #340 and #346 to storage configs.
See Also: #331

bpkroth and others added 30 commits May 4, 2023 14:31
@bpkroth
wip: json schema validation
8c0f03f
@bpkroth
optimizer schema rules
1344b5e
@bpkroth
wip
c79a767
@bpkroth
wip: schema validation connections and tests
41487e3
@bpkroth
wip
271d756
@bpkroth
split the mlos optimizer schema rules to a subschema file
c9d5a24
@bpkroth
wip: adding testing infrastructure for optimizer config schemas
e9f2e11
@bpkroth
Merge branch 'main' into json-schema-validation
826e45b
@bpkroth
add support for multiple class names and resolving the full one
b01f70a
@bpkroth
add schema matching rules for optimizers
ad4dcba
@bpkroth
Merge branch 'main' into json-schema-validation
07566f8
@bpkroth
back out changes to schema checking for unimplemented schema for now …
61c270f 
…- can stage more later
@bpkroth
borrow some code from another PR
144ddfa
@bpkroth
extend the mypy type-abstract ignores
444dd01
@bpkroth
remove $schema from the config if present
2fbd275
@bpkroth
fixups to schema validation with remote resolvers locally
5f5b2e0
@bpkroth
don't switch to multiple tunable values yet
0429047
@bpkroth
fixups to boolean parsing
707b064
@bpkroth
rely on schema validation and skip string conversion now
33a90f5
@bpkroth
Update mlos_bench/mlos_bench/config/optimizers/one_shot_opt.jsonc
c317cce
@bpkroth
whitespace
417344a
@bpkroth
fixup for lists of services configs
b4d19eb
@bpkroth
test fixups
33006a2
@bpkroth
more tests
2f47984
@bpkroth
fixups
f2c56d9
@bpkroth
test and schemas for tunable values
13c4db8
@bpkroth
more test cases and integration for tunable values schemas
a1ed8bd
@bpkroth
minor tweaks
4e1c8b0
@bpkroth
test fixups
0f5aa90
@bpkroth
rework tunable-values config to only accept a single set for now
c639b12
motus
motus reviewed May 15, 2023
View reviewed changes
Copy link
Member

@motus motus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Let's merge in the first PR and I'll do the final review

.pylintrc Outdated Show resolved Hide resolved
setup.cfg Outdated Show resolved Hide resolved
bpkroth and others added 14 commits May 15, 2023 21:37
@bpkroth
Merge branch 'main' into json-schema-validation-tunable-params
d9766f3
@bpkroth
Merge branch 'json-schema-validation-tunable-params' into json-schema…
e040969 
…-validation-storage
@bpkroth
fixup
83568b0
@bpkroth
Merge branch 'main' into json-schema-validation-tunable-params
a8a1c9c
@motus
Merge branch 'main' into json-schema-validation-tunable-params
77f1ce5
@bpkroth
applying some pr suggestions
07ace9b
@bpkroth
reorg tests to reduce duplication
2db4ac4
@bpkroth
Merge branch 'json-schema-validation-tunable-params' into json-schema…
b70624f 
…-validation-storage
@bpkroth
sync with prior PR changes
ffe1473
@bpkroth
make the test enumeration reproducible
5d77b95
@bpkroth
Merge branch 'json-schema-validation-tunable-params' into json-schema…
fac1cd3 
…-validation-storage
@bpkroth
make the test enumeration reproducible
6ba0645
@bpkroth
reuse extra config check
2ba0d3d
@bpkroth
Merge remote-tracking branch 'upstream/main' into json-schema-validat…
f970045 
…ion-storage
@bpkroth
Copy link
Contributor Author

bpkroth commented May 16, 2023

LGTM! Let's merge in the first PR and I'll do the final review

Previous PR is merged. Please have another look and approve when you get a sec. Mostly just a little more test code consolidation. Thanks!

@motus motus merged commit 3824f8e into microsoft:main May 16, 2023
@bpkroth bpkroth mentioned this pull request May 16, 2023
Merged
3 tasks
bpkroth added a commit that referenced this pull request May 17, 2023
@bpkroth
mlos_bench config json schema validation - cli (#352)
29c77c9 
Extends #340, #346, #349 to cli configs.
See Also: #331 

Also updates `--log_level` to support named logging levels (e.g. `INFO`)

- [x] add tests
  - [x] also backfilling tests of loading cli config examples (similar to what #313 did)
- [x] global config handling
@bpkroth bpkroth deleted the json-schema-validation-storage branch May 17, 2023 18:10
@bpkroth bpkroth mentioned this pull request May 19, 2023
Merged
3 tasks
bpkroth added a commit that referenced this pull request May 19, 2023
@bpkroth
mlos_bench config json schema validation - environments (#359)
0ff623b 
Extends #340, #346, #349, #352 to Environment configs.
See Also: #331 

- [x] main schema as a set of subschemas
- [x] connect schema validation in the mlos_bench code
- [x] test-cases
bpkroth added a commit that referenced this pull request May 22, 2023
@bpkroth
mlos_bench config json schema validation - services (#368)
f7348f1 
Extends #340, #346, #349, #352, #359 to Service configs.
See Also: #331 

- [x] main schema as a set of subschemas
- [x] connect schema validation in the mlos_bench code
- [x] test-cases

To do this we removed support for configs that were flat lists of service configs: `[ {"class": "service.class"} ]` and turned those into objects that have a single key `"services"` with the list instead. `{ "services": [ ... ] }`
This makes sure that all configs are dicts/objects that we can optionally put `"$schema"` attributes on but also simplifies some of the config loading logic.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@motus motus motus approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bpkroth @motus