hyperparameter-optimization-for-machine-learning.ipynb

{"cells":[{"source":"<a href=\"https://www.kaggle.com/code/faressayah/hyperparameter-optimization-for-machine-learning?scriptVersionId=118252459\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown","outputs":[],"execution_count":0},{"cell_type":"markdown","id":"256a54ca","metadata":{"papermill":{"duration":0.004214,"end_time":"2023-02-05T05:59:39.70826","exception":false,"start_time":"2023-02-05T05:59:39.704046","status":"completed"},"tags":[]},"source":["# 🎯Hyperparameter Optimization for Machine Learning\n","\n","The aim of this notebook:\n","> - Discuss multiple ways to optimize hyperparameters.\n","> - Understand the logic of each technique.\n","> - Considerations when utilizing each technique.\n","> - Master the use of Python open-source for hyperparameter tuning.\n","\n","---\n","\n","## Parameters in ML models\n","> - The objective of a typical learning algorithm is to find a function `f` that minimizes a certain `loss` over a `dataset`.\n","> - The learning algorithm produces `f` through the optimization of a training criteron with respect to a set of `parameters`.\n","\n","---\n","\n","## Hyperparameters in ML models\n","> - Hyperparameters are parameters that are not directly learnt by the learning algorithm.\n","> - Hyperparameters are specified outside of the training procedure.\n","> - Hyperparameters control the capacity of the model, i.e., how flexible the model is to fit the data.\n","> - Prevent over-fitting.\n","> - Hyperparameters could have a big impact on the performance of the learning algorithm.\n","> - Optimal hyperparameter settings often differ for different datasets. Therefore they should be optimized for each dataset.\n","---\n","\n","## Hyperparameter Nature\n",">- Some hyperparameters are discrete: Number of estimators in ensemble models.\n",">- Some hyperparameters are continuous: Penalization coefficient, Number of samples per split.\n",">- Some hyperparameters are categorical: Loss (deviance, exponential), Regularization (Lasso, Ridge)\n","\n","---\n","\n","## Parameters vs Hyperparameters\n","\n","|Parameters                  |   Hyperparameters |\n","|:-------------------------|----------------------:|\n","| - Intrinsic to model equation     | - Defined before training |\n","| - Optimized during training | - Constrain the algorithm|\n","\n","> - The process of finding the best Hyperparameters for a given dataset is called `Hyperparameter Tuning` or `Hyperparameter Optimization`.\n","\n","---\n","\n","## Challenges\n",">- We can't define a formula to find the hyperparameters.\n",">- Try different combinations of hyperparameters and evaluate model performance. The critical step is to choose how many different combinations we are going to test.\n","\n","The number of hyperparameter combination ---> the chance to get a better model ---> Computational cost\n","\n",">- How do we find the hyperparameter combinations to maximize performance while diminishing computational costs?\n","\n","---\n","\n","## Methods\n","Different hyperparamete optimization strategies:\n",">- Manual Search\n",">- Grid Search\n",">- Random Search\n",">- Bayesian Optimization\n","\n","---\n","\n","## Generalization vs Over-fitting\n","> Generalization is the ability of an algorithm to be effective across various inputs. The performance of the machine learning model is constant across different datasets (with the same distribution on the training data). When the model performs well on the train set, but not on new / naive data, the model over-fits to the training data.\n","\n","---\n","\n","## Training a Machine Learning Model\n","> To prevent over-fitting, it is common practice to:\n","> - Separate the data into a train and a test set.\n","> - Train the model in the train set.\n","> - Evaluate in the test set."]},{"cell_type":"markdown","id":"36bc46d3","metadata":{"papermill":{"duration":0.003313,"end_time":"2023-02-05T05:59:39.714988","exception":false,"start_time":"2023-02-05T05:59:39.711675","status":"completed"},"tags":[]},"source":["# Hyperparameter Optimization\n","\n","## Grid Search\n","\n",">- Exhaustive search through a specified subset of hyperparameters of a learning algorithm.\n",">- Examines all possible combinations of the specified hyperparameters (Cartesian product of hyperparameters).\n","\n","### Limitations\n",">- Curse of dimentionality: possible combinations grow exponentially with the number of hyperparameters.\n",">- Computationally expensive.\n",">- Hyperparameter values are determined manually.\n",">- Not ideal for continuous hyperparameters.\n",">- Does not explore the entire hyperparameter space (not feasible).\n",">- It performs worse than other searches (for models with complex hyperparameter spaces).\n","\n","### Advantages\n",">- For models with simpler hyperparameter spaces works well.\n",">- It can be parallelized.\n","\n","Grid Search is the most expensive method in terms of total computation time. However, if run in parallel, it is fast in terms of wall clock time. Sometimes, we run a small grid, determine where the optimum lies, and then expand the grid in that direction."]},{"cell_type":"markdown","id":"251aed5a","metadata":{"papermill":{"duration":0.002603,"end_time":"2023-02-05T05:59:39.720723","exception":false,"start_time":"2023-02-05T05:59:39.71812","status":"completed"},"tags":[]},"source":["# Tuning XGBoost with Optuna\n","\n","In this example, the hyperparameters `learning_rate`, `max_depth`, `n_estimators`, and `min_child_weight` are tuned using the Optuna library. The objective function is defined to return the negative accuracy on the test set, as Optuna minimizes the objective function. The `study.optimize` function is used to run the hyperparameter tuning, with `n_trials` specifying the number of trials to run. The final performance of the tuned classifier is evaluated on the test set."]},{"cell_type":"code","execution_count":1,"id":"44821ec6","metadata":{"execution":{"iopub.execute_input":"2023-02-05T05:59:39.728911Z","iopub.status.busy":"2023-02-05T05:59:39.728373Z","iopub.status.idle":"2023-02-05T06:03:54.73301Z","shell.execute_reply":"2023-02-05T06:03:54.731347Z"},"papermill":{"duration":255.013475,"end_time":"2023-02-05T06:03:54.737031","exception":false,"start_time":"2023-02-05T05:59:39.723556","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["\u001b[32m[I 2023-02-05 05:59:41,508]\u001b[0m A new study created in memory with name: no-name-19fb766a-50db-4e3f-be80-cb6db7ca8911\u001b[0m\n","\u001b[32m[I 2023-02-05 05:59:43,974]\u001b[0m Trial 0 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.04751609063782706, 'max_depth': 3, 'n_estimators': 615, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 05:59:45,901]\u001b[0m Trial 1 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.005501317485289229, 'max_depth': 5, 'n_estimators': 394, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 05:59:49,232]\u001b[0m Trial 2 finished with value: 0.04385964912280704 and parameters: {'learning_rate': 0.04092574159565932, 'max_depth': 4, 'n_estimators': 825, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 05:59:52,792]\u001b[0m Trial 3 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.008423628294485105, 'max_depth': 3, 'n_estimators': 852, 'min_child_weight': 4}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 05:59:53,997]\u001b[0m Trial 4 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.005887264005901771, 'max_depth': 4, 'n_estimators': 267, 'min_child_weight': 4}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 05:59:56,445]\u001b[0m Trial 5 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.0034453954507042274, 'max_depth': 6, 'n_estimators': 517, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:00,825]\u001b[0m Trial 6 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.05148991292028763, 'max_depth': 4, 'n_estimators': 846, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:03,090]\u001b[0m Trial 7 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.05210044406044372, 'max_depth': 6, 'n_estimators': 571, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:05,248]\u001b[0m Trial 8 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.08336290211668461, 'max_depth': 4, 'n_estimators': 570, 'min_child_weight': 5}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:05,892]\u001b[0m Trial 9 finished with value: 0.04385964912280704 and parameters: {'learning_rate': 0.09070148432787721, 'max_depth': 4, 'n_estimators': 151, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:08,519]\u001b[0m Trial 10 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.07034588183656731, 'max_depth': 7, 'n_estimators': 698, 'min_child_weight': 3}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:09,600]\u001b[0m Trial 11 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.028412204819186333, 'max_depth': 3, 'n_estimators': 255, 'min_child_weight': 4}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:11,063]\u001b[0m Trial 12 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.02595685437183739, 'max_depth': 3, 'n_estimators': 361, 'min_child_weight': 4}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:11,583]\u001b[0m Trial 13 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.022407402672227485, 'max_depth': 5, 'n_estimators': 104, 'min_child_weight': 3}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:15,292]\u001b[0m Trial 14 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.06516382327149618, 'max_depth': 3, 'n_estimators': 984, 'min_child_weight': 5}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:17,063]\u001b[0m Trial 15 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.036829217217893256, 'max_depth': 4, 'n_estimators': 435, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:19,774]\u001b[0m Trial 16 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.015609560270591783, 'max_depth': 3, 'n_estimators': 680, 'min_child_weight': 4}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:21,058]\u001b[0m Trial 17 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.0009590234675633967, 'max_depth': 5, 'n_estimators': 268, 'min_child_weight': 3}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:23,940]\u001b[0m Trial 18 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.016052373403744435, 'max_depth': 5, 'n_estimators': 687, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:25,018]\u001b[0m Trial 19 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.03582322091758394, 'max_depth': 4, 'n_estimators': 262, 'min_child_weight': 5}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:27,126]\u001b[0m Trial 20 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.04483747932664911, 'max_depth': 6, 'n_estimators': 508, 'min_child_weight': 3}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:29,950]\u001b[0m Trial 21 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.011020970649229992, 'max_depth': 6, 'n_estimators': 633, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:33,040]\u001b[0m Trial 22 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.020176888067447427, 'max_depth': 7, 'n_estimators': 484, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:37,037]\u001b[0m Trial 23 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.0021497479523405055, 'max_depth': 6, 'n_estimators': 762, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:38,681]\u001b[0m Trial 24 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.010461915122376661, 'max_depth': 7, 'n_estimators': 350, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:41,081]\u001b[0m Trial 25 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.029933319675883346, 'max_depth': 3, 'n_estimators': 603, 'min_child_weight': 3}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:43,729]\u001b[0m Trial 26 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.0004385753704199409, 'max_depth': 6, 'n_estimators': 514, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:44,571]\u001b[0m Trial 27 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.01502124080914382, 'max_depth': 4, 'n_estimators': 180, 'min_child_weight': 4}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:46,471]\u001b[0m Trial 28 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.02112846949332459, 'max_depth': 5, 'n_estimators': 437, 'min_child_weight': 2}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:48,080]\u001b[0m Trial 29 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.008363464721591258, 'max_depth': 5, 'n_estimators': 296, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:50,113]\u001b[0m Trial 30 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.007509208919489074, 'max_depth': 3, 'n_estimators': 451, 'min_child_weight': 1}. Best is trial 0 with value: 0.01754385964912286.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:53,223]\u001b[0m Trial 31 finished with value: 0.00877192982456143 and parameters: {'learning_rate': 0.015849264733013, 'max_depth': 5, 'n_estimators': 743, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:00:56,891]\u001b[0m Trial 32 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.00531981719038202, 'max_depth': 5, 'n_estimators': 804, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:00,799]\u001b[0m Trial 33 finished with value: 0.00877192982456143 and parameters: {'learning_rate': 0.015159362420230388, 'max_depth': 6, 'n_estimators': 927, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:05,609]\u001b[0m Trial 34 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.030738535265704102, 'max_depth': 4, 'n_estimators': 963, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:09,395]\u001b[0m Trial 35 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.016380494875550294, 'max_depth': 4, 'n_estimators': 923, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:12,456]\u001b[0m Trial 36 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.02519409042399323, 'max_depth': 5, 'n_estimators': 757, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:16,444]\u001b[0m Trial 37 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.011316412364003096, 'max_depth': 6, 'n_estimators': 898, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:19,864]\u001b[0m Trial 38 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.04359449812342037, 'max_depth': 4, 'n_estimators': 885, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:23,291]\u001b[0m Trial 39 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.03387009371191624, 'max_depth': 6, 'n_estimators': 810, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:26,228]\u001b[0m Trial 40 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.054842953835736785, 'max_depth': 5, 'n_estimators': 747, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:28,856]\u001b[0m Trial 41 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.004837146854093798, 'max_depth': 7, 'n_estimators': 554, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:32,220]\u001b[0m Trial 42 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.005515833948269306, 'max_depth': 6, 'n_estimators': 665, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:35,887]\u001b[0m Trial 43 finished with value: 0.00877192982456143 and parameters: {'learning_rate': 0.020796486830948483, 'max_depth': 6, 'n_estimators': 618, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:38,326]\u001b[0m Trial 44 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.024646947382656638, 'max_depth': 6, 'n_estimators': 601, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:41,138]\u001b[0m Trial 45 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.019782665985688116, 'max_depth': 3, 'n_estimators': 712, 'min_child_weight': 5}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:43,937]\u001b[0m Trial 46 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.013197020768332682, 'max_depth': 7, 'n_estimators': 630, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:47,508]\u001b[0m Trial 47 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.01904470475542719, 'max_depth': 5, 'n_estimators': 854, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:50,350]\u001b[0m Trial 48 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.02845309771684993, 'max_depth': 3, 'n_estimators': 732, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:53,546]\u001b[0m Trial 49 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.023376849048025235, 'max_depth': 4, 'n_estimators': 786, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:56,326]\u001b[0m Trial 50 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.01357143401085159, 'max_depth': 5, 'n_estimators': 650, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:01:58,177]\u001b[0m Trial 51 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.009800772888955658, 'max_depth': 6, 'n_estimators': 398, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:00,955]\u001b[0m Trial 52 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.0038984558698317243, 'max_depth': 6, 'n_estimators': 586, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:03,418]\u001b[0m Trial 53 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.0005018491187439005, 'max_depth': 6, 'n_estimators': 525, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:04,389]\u001b[0m Trial 54 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.007414522185125558, 'max_depth': 7, 'n_estimators': 176, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:07,542]\u001b[0m Trial 55 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.01773138853471269, 'max_depth': 6, 'n_estimators': 475, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:09,012]\u001b[0m Trial 56 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.01037869509804362, 'max_depth': 4, 'n_estimators': 320, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:11,461]\u001b[0m Trial 57 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.016350997613853804, 'max_depth': 6, 'n_estimators': 546, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:15,387]\u001b[0m Trial 58 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.022257345291573805, 'max_depth': 5, 'n_estimators': 999, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:18,910]\u001b[0m Trial 59 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.013743466385829767, 'max_depth': 7, 'n_estimators': 844, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:19,922]\u001b[0m Trial 60 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.026223224246012835, 'max_depth': 3, 'n_estimators': 225, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:22,790]\u001b[0m Trial 61 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.017714909469030128, 'max_depth': 5, 'n_estimators': 688, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:25,730]\u001b[0m Trial 62 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.0032010764900314292, 'max_depth': 5, 'n_estimators': 619, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:28,716]\u001b[0m Trial 63 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.012461352576854648, 'max_depth': 5, 'n_estimators': 723, 'min_child_weight': 5}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:29,332]\u001b[0m Trial 64 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.009621057122028216, 'max_depth': 6, 'n_estimators': 118, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:32,274]\u001b[0m Trial 65 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.006989752210187831, 'max_depth': 4, 'n_estimators': 661, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:35,422]\u001b[0m Trial 66 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.021857201890311765, 'max_depth': 5, 'n_estimators': 776, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:39,144]\u001b[0m Trial 67 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.014660014124673167, 'max_depth': 6, 'n_estimators': 573, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:40,933]\u001b[0m Trial 68 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.0010020727055596416, 'max_depth': 3, 'n_estimators': 401, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:44,574]\u001b[0m Trial 69 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.03790892685463157, 'max_depth': 4, 'n_estimators': 956, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:47,137]\u001b[0m Trial 70 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.03241369185445371, 'max_depth': 6, 'n_estimators': 643, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:48,799]\u001b[0m Trial 71 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.00906668449750316, 'max_depth': 7, 'n_estimators': 332, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:49,966]\u001b[0m Trial 72 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.004876752352833521, 'max_depth': 7, 'n_estimators': 232, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:51,627]\u001b[0m Trial 73 finished with value: 0.04385964912280704 and parameters: {'learning_rate': 0.01870071192281369, 'max_depth': 5, 'n_estimators': 374, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:52,953]\u001b[0m Trial 74 finished with value: 0.03508771929824561 and parameters: {'learning_rate': 0.016227379614285713, 'max_depth': 7, 'n_estimators': 284, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:55,178]\u001b[0m Trial 75 finished with value: 0.04385964912280704 and parameters: {'learning_rate': 0.012849777912360843, 'max_depth': 5, 'n_estimators': 503, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:02:58,068]\u001b[0m Trial 76 finished with value: 0.04385964912280704 and parameters: {'learning_rate': 0.027490247583713563, 'max_depth': 6, 'n_estimators': 693, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:00,964]\u001b[0m Trial 77 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.006610698348627916, 'max_depth': 7, 'n_estimators': 613, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:03,341]\u001b[0m Trial 78 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.010752077906057977, 'max_depth': 5, 'n_estimators': 550, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:04,472]\u001b[0m Trial 79 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.0026067686518941972, 'max_depth': 4, 'n_estimators': 232, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:07,542]\u001b[0m Trial 80 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.020766163445907836, 'max_depth': 3, 'n_estimators': 679, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:08,897]\u001b[0m Trial 81 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.015658809194078274, 'max_depth': 4, 'n_estimators': 186, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:09,798]\u001b[0m Trial 82 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.024752957779385416, 'max_depth': 4, 'n_estimators': 196, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:11,315]\u001b[0m Trial 83 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.011854776588672577, 'max_depth': 5, 'n_estimators': 323, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:12,096]\u001b[0m Trial 84 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.007678854731864032, 'max_depth': 6, 'n_estimators': 162, 'min_child_weight': 4}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:12,767]\u001b[0m Trial 85 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.019959115470564878, 'max_depth': 4, 'n_estimators': 145, 'min_child_weight': 5}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:13,687]\u001b[0m Trial 86 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.015197043710024139, 'max_depth': 3, 'n_estimators': 207, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:15,860]\u001b[0m Trial 87 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.005408647249894198, 'max_depth': 4, 'n_estimators': 472, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:17,116]\u001b[0m Trial 88 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.009354343076735123, 'max_depth': 5, 'n_estimators': 261, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:20,081]\u001b[0m Trial 89 finished with value: 0.052631578947368474 and parameters: {'learning_rate': 0.00013772833362326806, 'max_depth': 6, 'n_estimators': 589, 'min_child_weight': 1}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:21,755]\u001b[0m Trial 90 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.012118625147925242, 'max_depth': 7, 'n_estimators': 352, 'min_child_weight': 3}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:24,648]\u001b[0m Trial 91 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.023376701212057878, 'max_depth': 5, 'n_estimators': 713, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:27,711]\u001b[0m Trial 92 finished with value: 0.00877192982456143 and parameters: {'learning_rate': 0.017750033913365406, 'max_depth': 5, 'n_estimators': 742, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:30,845]\u001b[0m Trial 93 finished with value: 0.00877192982456143 and parameters: {'learning_rate': 0.017228030286640765, 'max_depth': 5, 'n_estimators': 752, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:34,218]\u001b[0m Trial 94 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.021104504700205383, 'max_depth': 5, 'n_estimators': 809, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:37,479]\u001b[0m Trial 95 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.017821975698531535, 'max_depth': 5, 'n_estimators': 785, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:41,594]\u001b[0m Trial 96 finished with value: 0.02631578947368418 and parameters: {'learning_rate': 0.014016306107532111, 'max_depth': 5, 'n_estimators': 734, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:44,996]\u001b[0m Trial 97 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.029132239127037014, 'max_depth': 5, 'n_estimators': 835, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:48,566]\u001b[0m Trial 98 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.003120429173892273, 'max_depth': 6, 'n_estimators': 750, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:51,393]\u001b[0m Trial 99 finished with value: 0.01754385964912286 and parameters: {'learning_rate': 0.017085789102666327, 'max_depth': 5, 'n_estimators': 673, 'min_child_weight': 2}. Best is trial 31 with value: 0.00877192982456143.\u001b[0m\n"]},{"name":"stdout","output_type":"stream","text":["Best hyperparameters:  {'learning_rate': 0.015849264733013, 'max_depth': 5, 'n_estimators': 743, 'min_child_weight': 2}\n","Best score:  0.9912280701754386\n","Test set accuracy:  1.0\n"]}],"source":["import numpy as np\n","import pandas as pd\n","from sklearn.datasets import load_breast_cancer\n","from sklearn.model_selection import train_test_split\n","from xgboost import XGBClassifier\n","import optuna\n","\n","# Load the breast cancer dataset\n","data = load_breast_cancer()\n","X = data.data\n","y = data.target\n","\n","# Split the data into training and testing sets\n","X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n","\n","def objective(trial):\n","    # Define the hyperparameters to tune\n","    learning_rate = trial.suggest_float(\"learning_rate\", 1e-5, 1e-1)\n","    max_depth = trial.suggest_int(\"max_depth\", 3, 7)\n","    n_estimators = trial.suggest_int(\"n_estimators\", 100, 1000)\n","    min_child_weight = trial.suggest_int(\"min_child_weight\", 1, 5)\n","    \n","    # Create an XGBoost classifier\n","    clf = XGBClassifier(\n","        learning_rate=learning_rate, \n","        max_depth=max_depth,\n","        n_estimators=n_estimators, \n","        min_child_weight=min_child_weight\n","    )\n","    \n","    # Train the classifier and calculate the accuracy on the validation set\n","    clf.fit(X_train, y_train)\n","    score = clf.score(X_test, y_test)\n","    \n","    return 1.0 - score\n","\n","# Use Optuna to tune the hyperparameters\n","study = optuna.create_study()\n","study.optimize(objective, n_trials=100)\n","\n","# Print the best hyperparameters and the best score\n","print(\"Best hyperparameters: \", study.best_params)\n","print(\"Best score: \", 1.0 - study.best_value)\n","\n","# Train the classifier with the best hyperparameters on the full training set\n","best_params = study.best_params\n","clf = XGBClassifier(\n","    learning_rate=best_params[\"learning_rate\"], \n","    max_depth=best_params[\"max_depth\"],\n","    n_estimators=best_params[\"n_estimators\"], \n","    min_child_weight=best_params[\"min_child_weight\"]\n",")\n","clf.fit(X, y)\n","\n","# Evaluate the tuned classifier on the test set\n","score = clf.score(X_test, y_test)\n","print(\"Test set accuracy: \", score)"]},{"cell_type":"markdown","id":"e88cfa5c","metadata":{"papermill":{"duration":0.00903,"end_time":"2023-02-05T06:03:54.755719","exception":false,"start_time":"2023-02-05T06:03:54.746689","status":"completed"},"tags":[]},"source":["# Tuning Random Forest with Optuna\n","\n","In this example, the hyperparameters `n_estimators`, `max_depth`, `min_samples_split`, and `min_samples_leaf` are tuned using the Optuna library. The objective function is defined to return the negative accuracy on the test set, as Optuna minimizes the objective function. The `study.optimize` function is used to run the hyperparameter tuning, with `n_trials` specifying the number of trials to run. The final performance of the tuned classifier is evaluated on"]},{"cell_type":"code","execution_count":2,"id":"ffb4518e","metadata":{"execution":{"iopub.execute_input":"2023-02-05T06:03:54.778465Z","iopub.status.busy":"2023-02-05T06:03:54.777149Z","iopub.status.idle":"2023-02-05T06:06:01.563678Z","shell.execute_reply":"2023-02-05T06:06:01.562536Z"},"papermill":{"duration":126.801533,"end_time":"2023-02-05T06:06:01.566578","exception":false,"start_time":"2023-02-05T06:03:54.765045","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["\u001b[32m[I 2023-02-05 06:03:54,996]\u001b[0m A new study created in memory with name: no-name-f260eda6-5c3e-4e9a-9219-befd239a1efc\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:56,808]\u001b[0m Trial 0 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 940, 'max_depth': 3, 'min_samples_split': 3, 'min_samples_leaf': 5}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:57,341]\u001b[0m Trial 1 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 261, 'max_depth': 4, 'min_samples_split': 4, 'min_samples_leaf': 5}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:57,904]\u001b[0m Trial 2 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 269, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 5}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:58,807]\u001b[0m Trial 3 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 413, 'max_depth': 6, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:03:59,996]\u001b[0m Trial 4 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 563, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:00,216]\u001b[0m Trial 5 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 101, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:00,517]\u001b[0m Trial 6 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 138, 'max_depth': 5, 'min_samples_split': 2, 'min_samples_leaf': 2}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:02,566]\u001b[0m Trial 7 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 982, 'max_depth': 6, 'min_samples_split': 5, 'min_samples_leaf': 4}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:02,890]\u001b[0m Trial 8 finished with value: 0.052631578947368474 and parameters: {'n_estimators': 152, 'max_depth': 4, 'min_samples_split': 3, 'min_samples_leaf': 1}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:04,037]\u001b[0m Trial 9 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 559, 'max_depth': 4, 'min_samples_split': 5, 'min_samples_leaf': 5}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:05,971]\u001b[0m Trial 10 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 979, 'max_depth': 3, 'min_samples_split': 2, 'min_samples_leaf': 3}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:07,419]\u001b[0m Trial 11 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 744, 'max_depth': 3, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:09,000]\u001b[0m Trial 12 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 734, 'max_depth': 7, 'min_samples_split': 3, 'min_samples_leaf': 3}. Best is trial 0 with value: 0.03508771929824561.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:10,746]\u001b[0m Trial 13 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 790, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:12,352]\u001b[0m Trial 14 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 834, 'max_depth': 3, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:14,250]\u001b[0m Trial 15 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 876, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:15,661]\u001b[0m Trial 16 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 686, 'max_depth': 4, 'min_samples_split': 5, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:17,501]\u001b[0m Trial 17 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 871, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:18,755]\u001b[0m Trial 18 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 639, 'max_depth': 3, 'min_samples_split': 3, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:20,423]\u001b[0m Trial 19 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 808, 'max_depth': 4, 'min_samples_split': 4, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:22,407]\u001b[0m Trial 20 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 947, 'max_depth': 5, 'min_samples_split': 5, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:23,388]\u001b[0m Trial 21 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 458, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:24,653]\u001b[0m Trial 22 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 593, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:25,629]\u001b[0m Trial 23 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 452, 'max_depth': 5, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:27,512]\u001b[0m Trial 24 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 783, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:29,409]\u001b[0m Trial 25 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 923, 'max_depth': 4, 'min_samples_split': 4, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:30,515]\u001b[0m Trial 26 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 501, 'max_depth': 7, 'min_samples_split': 3, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:31,956]\u001b[0m Trial 27 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 672, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:32,773]\u001b[0m Trial 28 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 370, 'max_depth': 7, 'min_samples_split': 2, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:33,746]\u001b[0m Trial 29 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 497, 'max_depth': 3, 'min_samples_split': 4, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:34,504]\u001b[0m Trial 30 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 349, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:35,803]\u001b[0m Trial 31 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 615, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:37,737]\u001b[0m Trial 32 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 899, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:38,352]\u001b[0m Trial 33 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 274, 'max_depth': 4, 'min_samples_split': 3, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:39,465]\u001b[0m Trial 34 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 515, 'max_depth': 6, 'min_samples_split': 2, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:40,673]\u001b[0m Trial 35 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 549, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:42,287]\u001b[0m Trial 36 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 735, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:42,991]\u001b[0m Trial 37 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 310, 'max_depth': 6, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:43,528]\u001b[0m Trial 38 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 231, 'max_depth': 7, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:44,249]\u001b[0m Trial 39 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 317, 'max_depth': 6, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:44,678]\u001b[0m Trial 40 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 180, 'max_depth': 7, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:45,570]\u001b[0m Trial 41 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 406, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:46,076]\u001b[0m Trial 42 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 228, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:48,101]\u001b[0m Trial 43 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 979, 'max_depth': 4, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:49,349]\u001b[0m Trial 44 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 569, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:51,162]\u001b[0m Trial 45 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 847, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:52,964]\u001b[0m Trial 46 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 846, 'max_depth': 7, 'min_samples_split': 5, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:54,642]\u001b[0m Trial 47 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 784, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:56,334]\u001b[0m Trial 48 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 802, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:57,805]\u001b[0m Trial 49 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 691, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:04:59,479]\u001b[0m Trial 50 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 781, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:01,414]\u001b[0m Trial 51 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 918, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:03,220]\u001b[0m Trial 52 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 847, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:05,324]\u001b[0m Trial 53 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 999, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:07,310]\u001b[0m Trial 54 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 942, 'max_depth': 7, 'min_samples_split': 5, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:08,771]\u001b[0m Trial 55 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 752, 'max_depth': 3, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:10,639]\u001b[0m Trial 56 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 872, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:12,581]\u001b[0m Trial 57 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 879, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:14,141]\u001b[0m Trial 58 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 710, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:15,900]\u001b[0m Trial 59 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 819, 'max_depth': 7, 'min_samples_split': 5, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:17,303]\u001b[0m Trial 60 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 654, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:19,192]\u001b[0m Trial 61 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 875, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:21,216]\u001b[0m Trial 62 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 959, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:22,877]\u001b[0m Trial 63 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 765, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:24,878]\u001b[0m Trial 64 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 916, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:26,623]\u001b[0m Trial 65 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 841, 'max_depth': 4, 'min_samples_split': 5, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:28,559]\u001b[0m Trial 66 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 893, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:30,117]\u001b[0m Trial 67 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 800, 'max_depth': 3, 'min_samples_split': 4, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:31,099]\u001b[0m Trial 68 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 450, 'max_depth': 6, 'min_samples_split': 3, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:32,601]\u001b[0m Trial 69 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 706, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:33,335]\u001b[0m Trial 70 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 312, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:34,397]\u001b[0m Trial 71 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 489, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:34,680]\u001b[0m Trial 72 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 115, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:35,050]\u001b[0m Trial 73 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 157, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:35,924]\u001b[0m Trial 74 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 400, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:36,375]\u001b[0m Trial 75 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 203, 'max_depth': 4, 'min_samples_split': 2, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:36,964]\u001b[0m Trial 76 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 261, 'max_depth': 6, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:38,981]\u001b[0m Trial 77 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 952, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:39,263]\u001b[0m Trial 78 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 109, 'max_depth': 7, 'min_samples_split': 3, 'min_samples_leaf': 5}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:39,946]\u001b[0m Trial 79 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 301, 'max_depth': 6, 'min_samples_split': 2, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:40,654]\u001b[0m Trial 80 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 347, 'max_depth': 3, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:42,448]\u001b[0m Trial 81 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 829, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:43,769]\u001b[0m Trial 82 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 586, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 1}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:45,347]\u001b[0m Trial 83 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 725, 'max_depth': 5, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:47,104]\u001b[0m Trial 84 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 856, 'max_depth': 4, 'min_samples_split': 3, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:48,426]\u001b[0m Trial 85 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 624, 'max_depth': 5, 'min_samples_split': 2, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:49,607]\u001b[0m Trial 86 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 536, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 2}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:51,236]\u001b[0m Trial 87 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 791, 'max_depth': 4, 'min_samples_split': 3, 'min_samples_leaf': 4}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:52,050]\u001b[0m Trial 88 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 375, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:52,876]\u001b[0m Trial 89 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 382, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:53,891]\u001b[0m Trial 90 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 464, 'max_depth': 7, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:54,654]\u001b[0m Trial 91 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 345, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:55,311]\u001b[0m Trial 92 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 281, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:56,099]\u001b[0m Trial 93 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 355, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:56,642]\u001b[0m Trial 94 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 238, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:57,197]\u001b[0m Trial 95 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 245, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:58,152]\u001b[0m Trial 96 finished with value: 0.02631578947368418 and parameters: {'n_estimators': 428, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:58,588]\u001b[0m Trial 97 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 182, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:59,242]\u001b[0m Trial 98 finished with value: 0.03508771929824561 and parameters: {'n_estimators': 291, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n","\u001b[32m[I 2023-02-05 06:05:59,543]\u001b[0m Trial 99 finished with value: 0.04385964912280704 and parameters: {'n_estimators': 122, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 3}. Best is trial 13 with value: 0.02631578947368418.\u001b[0m\n"]},{"name":"stdout","output_type":"stream","text":["Best hyperparameters:  {'n_estimators': 790, 'max_depth': 5, 'min_samples_split': 4, 'min_samples_leaf': 2}\n","Best score:  0.9736842105263158\n","Test set accuracy:  1.0\n"]}],"source":["from sklearn.ensemble import RandomForestClassifier\n","\n","# Load the breast cancer dataset\n","data = load_breast_cancer()\n","X = data.data\n","y = data.target\n","\n","# Split the data into training and testing sets\n","X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)\n","\n","def objective(trial):\n","    # Define the hyperparameters to tune\n","    n_estimators = trial.suggest_int(\"n_estimators\", 100, 1000)\n","    max_depth = trial.suggest_int(\"max_depth\", 3, 7)\n","    min_samples_split = trial.suggest_int(\"min_samples_split\", 2, 5)\n","    min_samples_leaf = trial.suggest_int(\"min_samples_leaf\", 1, 5)\n","    \n","    # Create a random forest classifier\n","    clf = RandomForestClassifier(\n","        n_estimators=n_estimators, \n","        max_depth=max_depth,\n","        min_samples_split=min_samples_split, \n","        min_samples_leaf=min_samples_leaf\n","    )\n","    \n","    # Train the classifier and calculate the accuracy on the validation set\n","    clf.fit(X_train, y_train)\n","    score = clf.score(X_test, y_test)\n","    \n","    return 1.0 - score\n","\n","# Use Optuna to tune the hyperparameters\n","study = optuna.create_study()\n","study.optimize(objective, n_trials=100)\n","\n","# Print the best hyperparameters and the best score\n","print(\"Best hyperparameters: \", study.best_params)\n","print(\"Best score: \", 1.0 - study.best_value)\n","\n","# Train the classifier with the best hyperparameters on the full training set\n","best_params = study.best_params\n","clf = RandomForestClassifier(\n","    n_estimators=best_params[\"n_estimators\"], \n","    max_depth=best_params[\"max_depth\"],\n","    min_samples_split=best_params[\"min_samples_split\"], \n","    min_samples_leaf=best_params[\"min_samples_leaf\"]\n",")\n","clf.fit(X, y)\n","\n","# Evaluate the tuned classifier on the test set\n","score = clf.score(X_test, y_test)\n","print(\"Test set accuracy: \", score)"]},{"cell_type":"markdown","id":"2fbbd92b","metadata":{"papermill":{"duration":0.015439,"end_time":"2023-02-05T06:06:01.598538","exception":false,"start_time":"2023-02-05T06:06:01.583099","status":"completed"},"tags":[]},"source":["## Random Search\n","\n",">- Hyperparameter values are selected by independent (random) draws from uniform distribution of the hyperparameter space. Random Search selects the combinations of hyperparameter values at random from all the possible combinations given a hyperparameter space.\n","\n","---\n","\n","## Random Search vs Grid Search\n",">- Some parameters affect performance a lot and some others don't (Low Effective Dimension). \n","\n","|      Random Search                                                |   Grid Search                            |\n","|:------------------------------------------------------------------|-----------------------------------------:|\n","| Allows the exploration of more dimensions of important parameters | Waste time exploring non-important dimensions |\n","| Select values from a distribution of parameter values             | Parameters are defined manually |\n","| Good for continuous parameters                                    | Good for discrete parameters |\n","\n","---\n","\n","## Considerations\n",">- We choose a (computational) budget independently of the number of parameters and possible values.\n",">- Adding parameters that do not influence the performance does not decrease efficiency of the search (if enough iterations are allowed).\n",">- Important to specify a continuous distribution of the hyperparameter to take full advantage of the randomization."]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.12"},"papermill":{"default_parameters":{},"duration":392.626621,"end_time":"2023-02-05T06:06:02.741994","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2023-02-05T05:59:30.115373","version":"2.3.4"}},"nbformat":4,"nbformat_minor":5}