Merge branch 'main' into hotflix/categorical-choice

Author: yhna940

README.md

@@ -16,15 +16,25 @@ SIATune is an open-source deep learning model hyperparameter tuning toolbox espe
   - [x] [MMSegmentation](https://github.com/open-mmlab/mmsegmentation)
   - [ ] [MMEditing](https://github.com/open-mmlab/mmediting)
 
-- **Support state-of-the-art hyperparameter tuning algorithms**
+- **Support hyperparameter search algorithms**
 
-    We provide state-of-the-art hyperparameter tuning algorithms such as below;
+  We provide hyperparameter search algorithms such as below;
   - [x] [Nevergrad](https://github.com/facebookresearch/nevergrad)
   - [x] [HyperOpt](https://github.com/hyperopt/hyperopt)
   - [x] [FLAML](https://github.com/microsoft/FLAML)
   - [ ] [Adaptive Experimentation (AX)](https://ax.dev/)
   - [ ] [Scikit-optimize](https://github.com/scikit-optimize/scikit-optimize)
 
+- **Schedule multiple experiments**
+
+  Various scheduling techniques are supported to efficiently manage many experiments.
+  - [x] [AsyncHyperBandScheduler](https://arxiv.org/abs/1810.05934)
+  - [ ] [PopulationBasedTraining](https://www.deepmind.com/blog/population-based-training-of-neural-networks)
+  - [ ] [MedianStoppingRule](https://research.google.com/pubs/pub46180.html)
+  - [ ] [Population Based Bandits](https://arxiv.org/abs/2002.02518)
+  - [ ] [HyperBandScheduler](https://arxiv.org/abs/1603.06560)
+
+
 - **Distributed tuning system based on Ray**
 
     Hyperparameter tuning with multi-GPU training or large-scale job scheduling are managed by Ray's distributed compute framework.

siatune/mm/tasks/cont_test_func.py

@@ -32,14 +32,17 @@ def _styblinksitang(x: np.ndarray, noise: float) -> float:
                  noise * np.random.normal(size=val.shape))
 
 
-def _step(s: float) -> float:
+def _step(s: float, eps: float = 1e-8) -> float:
     """Step function.
 
     Args:
         s (float): input.
+        eps (float): small number to avoid numerical errors.
     Returns:
         float: output.
     """
+    if s == 0:
+        s += eps
     return np.nan_to_num(np.exp(np.round(np.log(s))))
 
 
@@ -305,14 +308,17 @@ def stepdoublelinearslope(x: np.ndarray) -> float:
         return _step(np.abs(np.sum(x)))
 
     @staticmethod
-    def hm(x: np.ndarray) -> float:
+    def hm(x: np.ndarray, eps: float = 1e-8) -> float:
         """New multimodal function (proposed for Nevergrad).
 
         Args:
             x (np.ndarray): input vector.
+            eps (float): small number to avoid numerical errors.
         Returns:
             float: output.
         """
+        if not np.all(x):
+            x += eps
         return float((x**2).dot(1.1 + np.cos(1.0 / x)))
 
     @staticmethod