Refactor: Remove Cython - Reduce tech debt #346

eddiebergman · 2023-12-18T14:33:18Z

This PR attempts to first convert everything to Python, so a type checker can be run across everything as well as use linters properly. This also includes updating everything to pytest.

Note: This PR is unreviewable. The strategy has been to minimally change tests while refactoring to ensure tested behaviour still passes. Some notable exceptions are where functionality was removed or sampling strategy was changed and the expected values differ. Unfortunatly the formatter has rendered them also unreviewable.

Update: All SMAC unittests passed without code changes so it seems mostly backwards compatible.

Some notable changes:

No more unbounded Normals
No more quantized hyperparameters
Sampling is 3x faster, Neighbors 3.5x faster and validation 2x faster (see scripts/benchmark_sampling.py)
More code re-use between classes
Pure Python, jumping to definition and docstrings should work in editors
Defined data-structures to capture heirarchy in conditional spaces, moving the logic out of ConfigurationSpace.
Better defined public API for ConfigurationSpace.
Better support for custom hyperparameters, i.e. backed by Scipy Distributions (see BetaFloat/Int). Should require a lot less code to do so.
Better defined public API for hyperparameters
A whole load of deprecations with recommended alternatives
Categoricals/Ordinals support arbitrary objects, e.g. no more need to specify Category([True, False, "None"]) (Now handled in follow up feat: Allow arbitrary values as categoricals/constants/ordinals #376 )

TODO:

Seems deeply nested conditions with OR's were not solved Check that a condition is only part of one ConfigurationSpace object #72
We can add a flag Configuration(space, vector, validate=False) to deal with Problems integrating with BOHB #75
Allow constants with arbitrary values Boolean Constants #86 Legal values for hyperparameter #95

codecov · 2023-12-19T09:18:59Z

Codecov Report

Attention: Patch coverage is 92.06843% with 51 lines in your changes missing coverage. Please review.

Project coverage is 78.91%. Comparing base (93abc5b) to head (2a29120).
Report is 2 commits behind head on main.

❗ Current head 2a29120 differs from pull request most recent head 09beefd

Please upload reports for the commit 09beefd to get more accurate results.

Files	Patch %	Lines
ConfigSpace/forbidden.py	82.35%	17 Missing and 1 partial ⚠️
ConfigSpace/conditions.py	93.04%	7 Missing and 1 partial ⚠️
ConfigSpace/configuration_space.py	64.28%	3 Missing and 2 partials ⚠️
ConfigSpace/hyperparameters/normal_float.py	88.09%	4 Missing and 1 partial ⚠️
ConfigSpace/hyperparameters/normal_integer.py	85.71%	4 Missing and 1 partial ⚠️
ConfigSpace/hyperparameters/categorical.py	90.90%	3 Missing ⚠️
ConfigSpace/hyperparameters/ordinal.py	91.42%	3 Missing ⚠️
ConfigSpace/hyperparameters/uniform_integer.py	96.29%	2 Missing ⚠️
ConfigSpace/hyperparameters/numerical.py	97.05%	1 Missing ⚠️
ConfigSpace/hyperparameters/uniform_float.py	96.96%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #346      +/-   ##
==========================================
+ Coverage   73.54%   78.91%   +5.37%     
==========================================
  Files          29       45      +16     
  Lines        2831     4767    +1936     
  Branches      629     1005     +376     
==========================================
+ Hits         2082     3762    +1680     
- Misses        632      806     +174     
- Partials      117      199      +82

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

sarah-segel · 2024-06-24T14:02:59Z

FYI: I tested running DeepCave with ConfigSpace 0.8.0 and ran into the following problems when executing :

I run into an error when calling
from ConfigSpace.c_util import change_hp_value, check_forbidden
It seems like the default values in my configspace.json cause a problem when calling
from ConfigSpace.read_and_write import json as cs_json
cs_json.read(self.configspace_fn.read_text())
where I get:
__init__() got an unexpected keyword argument 'default'
When calling hp._transform_scalar(value) I get
AttributeError: 'UniformFloatHyperparameter' object has no attribute '_transform_scalar'

eddiebergman · 2024-06-27T16:15:56Z

@mfeurer, I did a major push on getting the documentation updated to mkdocs. You can view it locally by cloning this branch, pip install -e "[dev]" and running mkdocs serve. Likely another push should get this to the finish line.

@sarah-segel thanks for the report back, I will look into them with the next big push. Do you have any more information on the first one other than "ran into an error"? Second one is likely a bug on my behalf, will investigate. Last one is good to know about. It was a private method so I considered safe enough to remove. I will add it back in with the deprecation warning.

eddiebergman · 2024-07-01T11:51:04Z

@sarah-segel should have been fixed now, can you try again?

mfeurer

Only ten files to go

mfeurer · 2024-07-05T12:14:50Z

.github/workflows/docs.yml

+ - name: "Install dependancies"
+ run: python -m pip install -e ".[dev]"
+ - name: "Build Docs"
+ run: mkdocs build --clean --strict


Hey, is there a reason you removed the pushing of the documentation?

Yup, we moved away from Sphinx and I didn't know how to automate doc deployments given this setup. I will document how to do it in manual way in CONTRIBUTING.md

mfeurer · 2024-07-05T12:15:16Z

.github/workflows/pytest.yml

@@ -49,12 +47,6 @@ jobs:
 matrix:
 python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
 os: ["ubuntu-latest", "macos-latest", "windows-latest"]
- exclude:


Do this work now?

Apparently so

mfeurer · 2024-07-05T12:20:49Z

docs/reference/hyperparameters.md

+ ORDERABLE: ClassVar[bool] = True # Let ConfigSpace know there is an order to the values
+
+ alpha: float
+ """Some docstring decsription of this attribute."""


Should the docstring be further at the top of this class?

Nope, it's alright right here. It puts the documentation right with the declaration.

mfeurer · 2024-07-05T13:13:08Z

test/test_conditions.py

@@ -49,6 +49,10 @@
 from ConfigSpace.hyperparameters.hyperparameter import Hyperparameter


+def some_new_function():


What do we need this one for?

Good spot, removed

mfeurer · 2024-07-05T13:21:09Z

src/ConfigSpace/functional.py

+
+
+def center_range(
+ center: int,


What happens if the difference from the lower to the center is smaller than from the center to higher? Is this an issue?

Nope, roundrobin() will start just giving those from the above_center only after it can no longer yield equally between below_center and above_center

mfeurer · 2024-07-05T13:23:40Z

src/ConfigSpace/hyperparameters/hyperparameter.py

+ """Initialize a hyperparameter.
+
+ Args:
+ name:


Is it necessary to duplicate this information here?

Not entirely but I don't have it a different way. This is documenting parameters, while the other ones are documenting attributes. I would say it's necessary as it shows up in editors as help information

src/ConfigSpace/hyperparameters/hyperparameter.py

mfeurer · 2024-07-05T13:25:42Z

src/ConfigSpace/hyperparameters/hyperparameter.py

+ The probability density of the values. Where values are not legal,
+ the probability density is zero.
+ """
+ # TODO(eddiebergman): Backwards compatible restriction, why this restriction?


This would be great to know ;)

Will leave it in now unless it's a problem at some point

src/ConfigSpace/hyperparameters/hp_components.py

mfeurer

I finished looking at all files :) Now I need to look at your answer to my previous questions...

src/ConfigSpace/_condition_tree.py

src/ConfigSpace/hyperparameters/distributions.py

eddiebergman · 2024-07-12T12:32:31Z

Will be making a release once these tests pass :)

uncythonize, pytest update

ff0f2b2

This was referenced Dec 18, 2023

Remove hyperparameter in a configuration space #332

Open

Added TypedConfigurationSpace #345

Open

eddiebergman changed the title ~~Refactor: Update types for Cython~~ Refactor: Update types for Cython [In Progress] Dec 18, 2023

tests passing; fix add_conditions

e915964

eddiebergman added 18 commits December 19, 2023 14:58

Fixup tests

2a29120

refactor: UniformFloat

79decbd

optim: UniformFloat get neighbors

add84b5

refactor: deprecate UniformFloat is_legal_vector

a3f4297

big overhaul

60dc4db

fix: depercation wraning of np.integer for dtype

279cac2

refactor: Big internal rewrite, passing tests

3bc359f

test passing checkpoint

b7f5d9d

pythonize conditionals

fb368c4

optim: Topological sort and vectorize cond/forbiddens

e4064cf

optim: Further small optimizations

5ebd21f

optim: Transpose sampling matrix for cache locality

a4916be

optim: Add test script for neighbor sampling

4ab1c80

test: Repass tests

584b8ef

tmpcommmit - lets pray

87eb45d

refactor: Factor out DAG structure into dedicated class

fceffa6

refactor: Move things out of c_util

ba1cb59

fix: Minimum spanning

26430fd

eddiebergman changed the title ~~Refactor: Update types for Cython [In Progress]~~ Refactor: Remove Cython - Reduce tech debt Mar 28, 2024

eddiebergman mentioned this pull request Mar 28, 2024

Missing documentation on Forbidden Relations #352

Closed

eddiebergman added 2 commits March 28, 2024 16:16

fix: min-cond-span sorted by min(depth of all children)

5fb53f6

refactor: All tests and benchmarks pass

82e3fab

eddiebergman mentioned this pull request Apr 10, 2024

Importing a custom incomplete configuration space? #355

Closed

Merge branch 'main' into typing-cython-3

147e6ec

doc: Major doc overhaul to mkdocs

8a4ac0b

eddiebergman added 4 commits July 1, 2024 13:36

fix: Add backwards compatible check_forbidden

369803c

fix: Allow backwards compat "default" key in encoded spaces

5152f9e

fix: Backwards compatible private methods

cc5fd78

refactor: Use check_ instead of is_ as method raises

cf011bf

eddiebergman added 4 commits July 1, 2024 14:26

doc: Forbiddens

bc3afcd

test: fix test funcion name

2a48d08

fix(Forbidden): Uncomment value checking on ForbiddenEqualsClaus

995c417

ci: Update doc and test workflows

09beefd

eddiebergman mentioned this pull request Jul 4, 2024

ConfigSpace is not compatible with numpy 2.0 #367

Closed

mfeurer reviewed Jul 5, 2024

View reviewed changes

eddiebergman added 7 commits July 12, 2024 14:23

test: Remove stray function

debe675

doc: Type fix

897bec7

doc: Add note to _unsafe code in hp_components

b1f1fb9

doc: Typo fix

80e2c7c

doc: Improve code doc attribute clarity

d23879d

doc: Change note to todo

de0b36e

doc: Fix documentation for continuous_neighborhood

82d66fd

eddiebergman merged commit f4f4701 into main Jul 12, 2024
17 checks passed

bpkroth mentioned this pull request Aug 5, 2024

No more quantized hyperparameters? #390

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: Remove Cython - Reduce tech debt #346

Refactor: Remove Cython - Reduce tech debt #346

eddiebergman commented Dec 18, 2023 •

edited

Loading

codecov bot commented Dec 19, 2023 •

edited

Loading

sarah-segel commented Jun 24, 2024

eddiebergman commented Jun 27, 2024 •

edited

Loading

eddiebergman commented Jul 1, 2024

mfeurer left a comment

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer Jul 5, 2024

eddiebergman Jul 12, 2024

mfeurer left a comment

eddiebergman commented Jul 12, 2024

		@@ -49,6 +49,10 @@
		from ConfigSpace.hyperparameters.hyperparameter import Hyperparameter


		def some_new_function():

Refactor: Remove Cython - Reduce tech debt #346

Refactor: Remove Cython - Reduce tech debt #346

Conversation

eddiebergman commented Dec 18, 2023 • edited Loading

codecov bot commented Dec 19, 2023 • edited Loading

Codecov Report

sarah-segel commented Jun 24, 2024

eddiebergman commented Jun 27, 2024 • edited Loading

eddiebergman commented Jul 1, 2024

mfeurer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfeurer left a comment

Choose a reason for hiding this comment

eddiebergman commented Jul 12, 2024

eddiebergman commented Dec 18, 2023 •

edited

Loading

codecov bot commented Dec 19, 2023 •

edited

Loading

eddiebergman commented Jun 27, 2024 •

edited

Loading