Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API refactoring + API tests and checks #33

Merged
merged 3 commits into from
Nov 28, 2023
Merged

API refactoring + API tests and checks #33

merged 3 commits into from
Nov 28, 2023

Conversation

TarekAbouChakra
Copy link
Collaborator

@TarekAbouChakra TarekAbouChakra commented Nov 16, 2023

This PR focuses on NePS API refactoring and introduces several enhancements and tests:

  • YAML generated to save optimizer information is now better configured, with specific details on the optimizer's arguments and the type of search done.
  • Added test cases for each neps.run call type, covering Baseoptimizer, default YAML, and user YAML scenarios.
  • Introduced a Metahyper test for the optimizer, evaluating multi-thread processing and pausing/resuming run scenarios. This test ensures that users won't unintentionally use different searchers for the same run, providing a more stable and predictable optimization environment.
  • Added API tests on GitHub Actions.
  • Accepts None to be passed for the pipeline space when the searcher is of type BaseOptimizer. Raises an error otherwise

Copy link
Contributor

@eddiebergman eddiebergman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just left some minor comments, don't need to act on them really. I do like the idea that you test runnable examples through pytest, I think going forward this will make it easier to do integration tests and make test writing easier for people.

If you are happy with others using the examples, you could move them to the root, ./examples directory of the repo and still test them there. That way people will see them, use them, and their tested.

src/neps/api.py Outdated
@@ -106,7 +108,7 @@ def run(
ignore_errors: bool = False,
loss_value_on_error: None | float = None,
cost_value_on_error: None | float = None,
pre_load_hooks: List=[],
pre_load_hooks: List = [],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some information about typing, don't need to do anything here but thought it mgiht be good to share:

  1. If you have from __future__ import annotations (which there is), you can use the lowercase list
  2. Do not use mutable arguments as a default.
  3. We can type it better by following down where they're used, from what I gather it accepts Callable[[metahyper.Sampler], metahyper.Sampler], where it can modify the sampler. Consider the typing can happen at a neps level and we only care about class BaseOptimizer(metahyper.Sampler): ..., then the full signature might look like:
def run(
    ...,
    pre_load_hooks: list[Callable[[BaseOptimizer], BaseOptimizer]] | None = None
):
    if pre_load_hooks is None:
        pre_load_hooks = []
  1. We can actually improve it even further using typing.Iterable to mean anything that we can iterate
from typing import Iterable

def run(
    ...,
    pre_load_hooks: Iterable[Callable[[BaseOptimizer], BaseOptimizer]] | None = None
):
    if pre_load_hooks is None:
        pre_load_hooks = []

This means someone could pass a tuple, set, generator or really anything where we can go for x in my_iterable: ....

  1. Last bit is definitely over the top and low priority, but just to share some new typing tips. The function is actually a bit more specific in that it should return the same type of BaseOptimizer that went into it, not just any base optimizer. We can use TypeVar's for this, which are essentially C+++ Templates or Java Generics.
from typing import TypeVar

# This is some type that must be subclass of BaseOptimizer
OptimizerT = TypeVar("OptimizerT", bound=BaseOptimizer)

def run(
    ...,
    pre_load_hooks: Iterable[Callable[[OptimizerT], OptimizerT]] | None = None
):
    if pre_load_hooks is None:
        pre_load_hooks = []

@Neeratyoy thought you might also like to know about these things if you are reviewing

src/neps/api.py Outdated Show resolved Hide resolved
src/neps/api.py Outdated Show resolved Hide resolved
src/neps/utils/common.py Show resolved Hide resolved
tests/test_neps_api/test_api.py Show resolved Hide resolved
tests/test_neps_api/test_api.py Outdated Show resolved Hide resolved
@Neeratyoy Neeratyoy merged commit eec1222 into master Nov 28, 2023
11 checks passed
@TarekAbouChakra TarekAbouChakra deleted the api-checks branch November 29, 2023 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants