Implement ConfigIDs and rename objective function budget to fidelity

README.md

@@ -25,8 +25,8 @@ pip install -e DEHB  # -e stands for editable, lets you modify the code and reru
 To run PyTorch example: (*note additional requirements*) 
 ```bash
 python examples/03_pytorch_mnist_hpo.py \
-    --min_budget 1 \
-    --max_budget 3 \
+    --min_fidelity 1 \
+    --max_fidelity 3 \
     --runtime 60 \
     --verbose
 ```
@@ -62,8 +62,8 @@ to it by that DEHB run.
 To run the PyTorch MNIST example on a single node using 2 workers:  
 ```bash
 python examples/03_pytorch_mnist_hpo.py \
-    --min_budget 1 \
-    --max_budget 3 \
+    --min_fidelity 1 \
+    --max_fidelity 3 \
     --runtime 60 \
     --n_workers 2 \
     --single_node_with_gpus \
@@ -96,8 +96,8 @@ bash utils/run_dask_setup.sh \
 # Make sure to sleep to allow the workers to setup properly
 sleep 5
 python examples/03_pytorch_mnist_hpo.py \
-    --min_budget 1 \
-    --max_budget 3 \
+    --min_fidelity 1 \
+    --max_fidelity 3 \
     --runtime 60 \
     --scheduler_file dask_dump/scheduler.json \
     --verbose
@@ -111,9 +111,9 @@ and were found to be *generally* useful across all cases tested.
 However, the parameters are still available for tuning to a specific problem.
 
 The Hyperband components:
-* *min\_budget*: Needs to be specified for every DEHB instantiation and is used in determining 
-the budget spacing for the problem at hand.
-* *max\_budget*: Needs to be specified for every DEHB instantiation. Represents the full-budget 
+* *min\_fidelity*: Needs to be specified for every DEHB instantiation and is used in determining 
+the fidelity spacing for the problem at hand.
+* *max\_fidelity*: Needs to be specified for every DEHB instantiation. Represents the full-fidelity 
 evaluation or the actual black-box setting.
 * *eta*: (default=3) Sets the aggressiveness of Hyperband's aggressive early stopping by retaining
 1/eta configurations every round

docs/index.md

@@ -42,16 +42,16 @@ Next, we need an `object_function`, which we are aiming to optimize:
 ```python exec="true" source="material-block" result="python" title="Configuration Space" session="someid"
 import numpy as np
 
-def objective_function(x: Configuration, budget: float, **kwargs):
+def objective_function(x: Configuration, fidelity: float, **kwargs):
     # Replace this with your actual objective value (y) and cost.
-    cost = (10 if x["x1"] == "red" else 100) + budget
+    cost = (10 if x["x1"] == "red" else 100) + fidelity
     y = x["x0"] + np.random.uniform()
     return {"fitness": y, "cost": x["x0"]}
 
 sample_config = cs.sample_configuration()
 print(sample_config)
 
-result = objective_function(sample_config, budget=10)
+result = objective_function(sample_config, fidelity=10)
 print(result)
 ```
 
@@ -65,20 +65,20 @@ optimizer = DEHB(
     f=objective_function,
     cs=cs,
     dimensions=dim,
-    min_budget=3,
-    max_budget=27,
+    min_fidelity=3,
+    max_fidelity=27,
     eta=3,
     n_workers=1,
     output_path="./logs",
 )
 
 # Run optimization for 1 bracket. Output files will be saved to ./logs
 traj, runtime, history = optimizer.run(brackets=1, verbose=True)
-config, fitness, runtime, budget, _ = history[0]
+config, fitness, runtime, fidelity, _ = history[0]
 print("config", config)
 print("fitness", fitness)
 print("runtime", runtime)
-print("budget", budget)
+print("fidelity", fidelity)
 ```
 
 ### Running DEHB in a parallel setting
@@ -112,8 +112,8 @@ to it by that DEHB run.
 To run the PyTorch MNIST example on a single node using 2 workers:  
 ```bash
 python examples/03_pytorch_mnist_hpo.py \
-    --min_budget 1 \
-    --max_budget 3 \
+    --min_fidelity 1 \
+    --max_fidelity 3 \
     --runtime 60 \
     --n_workers 2 \
     --single_node_with_gpus \
@@ -147,8 +147,8 @@ bash utils/run_dask_setup.sh \
 sleep 5
 
 python examples/03_pytorch_mnist_hpo.py \
-    --min_budget 1 \
-    --max_budget 3 \
+    --min_fidelity 1 \
+    --max_fidelity 3 \
     --runtime 60 \
     --scheduler_file dask_dump/scheduler.json \
     --verbose

examples/00_interfacing_DEHB.ipynb

@@ -40,7 +40,7 @@
     "\n",
     "DEHB also uses Hyperband along with DE, to allow for cheaper approximations of the actual evaluations of $x$. Let $f(x)$ be the validation error of training a multilayer perceptron (MLP) on the complete training set. Multi-fidelity algorithms such as Hyperband, allow for cheaper approximations along a possible *fidelity*. For the MLP, a subset of the dataset maybe a cheaper approximation to the full data set evaluation. Whereas the fidelity can be quantifies as the fraction of the dataset used to evaluate the configuration $x$, instead of the full dataset. Such approximations can allow sneak-peek into the black-box, potentially revealing certain landscape feature of *f(x)*, thus rendering it a *gray*-box and not completely opaque and black! \n",
     "\n",
-    "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [budget_{min}, budget_{max}]$, then $f(x, budget_{max})$ would be equivalent to the black-box case of $f(x)$.\n",
+    "The $z$ parameter is the fidelity parameter to the black-box function. If $z \\in [fidelity_{min}, fidelity_{max}]$, then $f(x, fidelity_{max})$ would be equivalent to the black-box case of $f(x)$.\n",
     "\n",
     "![boxes](imgs/black-gray-box.png)"
    ]
@@ -62,15 +62,15 @@
    "source": [
     "def target_function(\n",
     "    x: Union[ConfigSpace.Configuration, List, np.array], \n",
-    "    budget: Union[int, float] = None,\n",
+    "    fidelity: Union[int, float] = None,\n",
     "    **kwargs\n",
     ") -> Dict:\n",
     "    \"\"\" Target/objective function to optimize\n",
     "    \n",
     "    Parameters\n",
     "    ----------\n",
     "    x : configuration that DEHB wants to evaluate\n",
-    "    budget : parameter determining cheaper evaluations\n",
+    "    fidelity : parameter determining cheaper evaluations\n",
     "    \n",
     "    Returns\n",
     "    -------\n",
@@ -83,7 +83,7 @@
     "    # remove the code snippet below\n",
     "    start = time.time()\n",
     "    y = np.random.uniform()  # placeholder response of evaluation\n",
-    "    time.sleep(budget)       # simulates runtime (mostly proportional to fidelity)\n",
+    "    time.sleep(fidelity)       # simulates runtime (mostly proportional to fidelity)\n",
     "    cost = time.time() - start\n",
     "    \n",
     "    # result dict passed to DE/DEHB as function evaluation output\n",
@@ -171,8 +171,9 @@
     {
      "data": {
       "text/plain": [
-       "Configuration:\n",
-       "  x0, Value: 3.716302229868112"
+       "Configuration(values={\n",
+       "  'x0': 8.107160631154175,\n",
+       "})"
       ]
      },
      "execution_count": 5,
@@ -198,7 +199,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Defining fidelity/budget range for the target function"
+    "### Defining fidelity range for the target function"
    ]
   },
   {
@@ -207,7 +208,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "min_budget, max_budget = (0.1, 3) "
+    "min_fidelity, max_fidelity = (0.1, 3) "
    ]
   },
   {
@@ -244,8 +245,8 @@
     "    f=target_function,\n",
     "    dimensions=dimensions,\n",
     "    cs=cs,\n",
-    "    min_budget=min_budget,\n",
-    "    max_budget=max_budget,\n",
+    "    min_fidelity=min_fidelity,\n",
+    "    max_fidelity=max_fidelity,\n",
     "    output_path=\"./temp\",\n",
     "    n_workers=1        # set to >1 to utilize parallel workers\n",
     ")\n",
@@ -281,9 +282,9 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Configuration:\n",
-      "  x0, Value: 4.060258498267547\n",
-      "\n"
+      "Configuration(values={\n",
+      "  'x0': 4.152073449922892,\n",
+      "})\n"
      ]
     }
    ],
@@ -308,14 +309,14 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "2021-10-22 14:45:56.117 | INFO     | dehb.optimizers.dehb:reset:107 - \n",
-      "\n",
-      "RESET at 10/22/21 14:45:56 CEST\n",
+      "\u001b[32m2023-10-22 20:03:06.057\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36mreset\u001b[0m:\u001b[36m121\u001b[0m - \u001b[1m\n",
       "\n",
+      "RESET at 10/22/23 20:03:06 CEST\n",
       "\n",
-      "(Configuration:\n",
-      "  x0, Value: 3.724555206841792\n",
-      ", 0.0938589687572785)\n"
+      "\u001b[0m\n",
+      "(Configuration(values={\n",
+      "  'x0': 8.96840263375364,\n",
+      "}), 0.05819975786653586)\n"
      ]
     }
    ],
@@ -343,14 +344,14 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "2021-10-22 14:45:58.567 | INFO     | dehb.optimizers.dehb:reset:107 - \n",
-      "\n",
-      "RESET at 10/22/21 14:45:58 CEST\n",
+      "\u001b[32m2023-10-22 20:03:11.073\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mdehb.optimizers.dehb\u001b[0m:\u001b[36mreset\u001b[0m:\u001b[36m121\u001b[0m - \u001b[1m\n",
       "\n",
+      "RESET at 10/22/23 20:03:11 CEST\n",
       "\n",
-      "(Configuration:\n",
-      "  x0, Value: 4.341818535733585\n",
-      ", 3.653636256717441e-05)\n"
+      "\u001b[0m\n",
+      "(Configuration(values={\n",
+      "  'x0': 8.708444163420975,\n",
+      "}), 0.0710929937087792)\n"
      ]
     }
    ],
@@ -381,9 +382,9 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "(Configuration:\n",
-      "  x0, Value: 4.610766436763522\n",
-      ", 0.007774399252232556)\n"
+      "(Configuration(values={\n",
+      "  'x0': 8.454086817115218,\n",
+      "}), 0.016305791635409683)\n"
      ]
     }
    ],
@@ -392,8 +393,8 @@
     "    f=target_function,\n",
     "    dimensions=dimensions,\n",
     "    cs=cs,\n",
-    "    min_budget=min_budget,\n",
-    "    max_budget=max_budget,\n",
+    "    min_fidelity=min_fidelity,\n",
+    "    max_fidelity=max_fidelity,\n",
     "    output_path=\"./temp\",\n",
     "    n_workers=2\n",
     ")\n",
@@ -413,9 +414,9 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Configuration:\n",
-      "  x0, Value: 4.610766436763522\n",
-      "\n"
+      "Configuration(values={\n",
+      "  'x0': 8.454086817115218,\n",
+      "})\n"
      ]
     }
    ],
@@ -432,10 +433,10 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "0.007774399252232556 0.007774399252232556\n",
-      "Configuration:\n",
-      "  x0, Value: 4.610766436763522\n",
-      "\n"
+      "0.016305791635409683 0.016305791635409683\n",
+      "Configuration(values={\n",
+      "  'x0': 8.454086817115218,\n",
+      "})\n"
      ]
     }
    ],
@@ -454,7 +455,7 @@
     "\n",
     "As detailed above, the problem definition needs to be input to DEHB as the following information:\n",
     "* the *target_function* (`f`) that is the primary black-box function to optimize\n",
-    "* the fidelity range of `min_budget` and `max_budget` that allows the cheaper, faster gray-box optimization of `f`\n",
+    "* the fidelity range of `min_fidelity` and `max_fidelity` that allows the cheaper, faster gray-box optimization of `f`\n",
     "* the search space or the input domain of the function `f`, that can be represented as a `ConfigSpace` object and passed to DEHB at initialization\n",
     "\n",
     "\n",
@@ -465,13 +466,20 @@
     "\n",
     "DEHB will terminate once its chosen runtime budget is exhausted, and report the incumbent found. DEHB, as an *anytime* algorithm, constantly writes to disk a lightweight `json` file with the best found configuration and its score seen till that point."
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "dask",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "dask"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -483,7 +491,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.6.9"
+   "version": "3.9.16"
   }
  },
  "nbformat": 4,