callback function error!

[Daniel-Trung-Nguyen]

I was trying to apply the early stopping example for a regression problem and got this error:

[ERROR] [2022-08-18 04:33:27,482:Client-AutoML(1):botanal] Cannot register callback of type <class 'function'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/automl.py", line 931, in fit
    _proc_smac.run_smbo()
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/smbo.py", line 496, in run_smbo
    smac.register_callback(self.trials_callback)
  File "/usr/local/lib/python3.7/dist-packages/smac/facade/smac_ac_facade.py", line 850, in register_callback
    raise ValueError("Cannot register callback of type %s" % type(callback))
ValueError: Cannot register callback of type <class 'function'>
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-30-b2f7d005a0d7>](https://localhost:8080/#) in <module>
      8                                                         get_trials_callback=callback
      9                                                     )
---> 10 automl.fit(X_train, y_train, dataset_name="botanal")

5 frames
[/usr/local/lib/python3.7/dist-packages/smac/facade/smac_ac_facade.py](https://localhost:8080/#) in register_callback(self, callback)
    848                 break
    849         if key is None:
--> 850             raise ValueError("Cannot register callback of type %s" % type(callback))
    851         self.solver._callbacks[key].append(callback)

ValueError: Cannot register callback of type <class 'function'>

from smac.optimizer.smbo import SMBO
from smac.runhistory.runhistory import RunInfo, RunValue
def callback(
    smbo: SMBO,
    run_info: RunInfo,
    result: RunValue,
    time_left: float):
    """Stop early if we get a very low cost value for a single run
    The return value indicates to SMAC whether to stop or not. False will
    stop the search process while any other value will mean it continues.
    """
    # You can find out the parameters in the SMAC documentation
    # https://automl.github.io/SMAC3/main/
    if result.cost <= 0.02:
        print("Stopping!")
        print(run_info)
        print(result)
        return False

automl = autosklearn.regression.AutoSklearnRegressor(time_left_for_this_task=120,
                                                        per_run_time_limit=30,
                                                        tmp_folder="/tmp/autosklearn_regression_example_tmp",
                                                        get_trials_callback=callback
                                                    )
automl.fit(X_train, y_train, dataset_name="botanal")

[eddiebergman]

Hi @Daniel-Trung-Nguyen,

I'll look into this but this feature is currently only in the documentation for development and only available in the development branch, are you using the development branch?

Best,
Eddie

[Daniel-Trung-Nguyen]

I just pulled the dev brand and tested the callback (on Colab).
I received this error:

Process pynisher function call:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 236, in _determine_key_type
    return dtype_to_str[type(key)]
KeyError: <class 'numpy.str_'>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/pynisher/limit_function_call.py", line 133, in subprocess_func
    return_value = ((func(*args, **kwargs), 0))
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/smbo.py", line 166, in _calculate_metafeatures_encoded
    logger=logger_,
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/metalearning/metafeatures/metafeatures.py", line 1122, in calculate_all_metafeatures_encoded_labels
    logger=logger,
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/metalearning/metafeatures/metafeatures.py", line 1194, in calculate_all_metafeatures
    X_transformed = DPP.fit_transform(X)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/base.py", line 699, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/usr/local/lib/python3.7/dist-packages/autosklearn/pipeline/components/data_preprocessing/feature_type.py", line 216, in fit
    self.column_transformer.fit(X, y)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 470, in fit
    self.fit_transform(X, y=y)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 505, in fit_transform
    self._validate_remainder(X)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 325, in _validate_remainder
    for cols in self._columns)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/compose/_column_transformer.py", line 325, in <genexpr>
    for cols in self._columns)
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 257, in _determine_key_type
    key_type = {_determine_key_type(elt) for elt in unique_key}
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 257, in <setcomp>
    key_type = {_determine_key_type(elt) for elt in unique_key}
  File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/__init__.py", line 238, in _determine_key_type
    raise ValueError(err_msg)
ValueError: No valid specification of the columns. Only a scalar, list or slice of all integers or all strings, or boolean mask is allowed
[WARNING] [2022-08-18 20:51:10,640:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:11,533:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:12,428:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:13,310:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:14,193:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:15,077:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:15,958:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:16,838:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:17,715:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:18,622:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:19,513:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:21,184:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:22,095:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:23,008:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:23,918:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:24,826:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:25,732:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:26,632:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:27,532:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:28,437:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:29,333:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:30,229:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:31,137:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:32,025:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:32,935:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:33,827:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:39,824:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:45,627:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:47,198:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:55,572:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:57,108:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:51:58,088:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:03,259:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:04,243:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:05,215:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:10,838:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:11,774:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:12,709:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:13,671:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:19,213:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:20,152:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:21,108:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:27,076:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:28,018:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:28,959:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:29,908:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:30,868:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:31,872:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:32,824:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:33,803:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:34,742:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:35,685:Client-EnsembleBuilder] No runs were available to build an ensemble from
[WARNING] [2022-08-18 20:52:36,635:Client-EnsembleBuilder] No runs were available to build an ensemble from

[eddiebergman]

@Daniel-Trung-Nguyen, this seems unrelated but at least it seems that the callback works as intended.

Can you provide a code snippet that reproduces this error, or a snippet of one row of the dataset or at least the feature types of your columns?

Best,
Eddie