AttributeError: 'DataFrame' object has no attribute 'append'

[kasvegas]

when running the frameworks on my benchmark suite (of two tasks) I am running into several issues...is there are current known issue executing the frameworks? running the single line "runbenchmark.py framework openml/s/###" with the available frameworks is either not executing at all, or in some cases executed fold 0 on the first task but never gets to the second task in the suite. The following error keeps occurring....

AttributeError: 'DataFrame' object has no attribute 'append'

Thank you again for this contribution, we are currently working on a research paper and comparing these frameworks with your tool will be extremely beneficial.

[PGijsbers]

I am not aware of any issues, if you can provide me with a MWE (which version are you running, what are the tasks, what command are you invoking, do you have custom configurations). Try to trim it down as much as possible to the out-of-the-box setup. With that info I might be able to help you further.

[kasvegas]

Thanks again for the prompt reply. We have not made any customizations. We have followed the set up instructions provided on the website. The commands we are invoking are the "benchmark.py framework openml/s/411". For most frameworks we get the following output below, I have provided the code and output for the randomforest. This same error also occurs on autoscikitlearn. For h2o and autogluon, it begins to work but seems to error after fold 0.

kasvegas@Ubuntu:~/automlbenchmark$ python3 runbenchmark.py randomforest openml/s/411

Running benchmark `randomforest` on `openml/s/411` framework in `local` mode.

Loading frameworks definitions from ['/home/kasvegas/automlbenchmark/resources/frameworks.yaml'].

Loading benchmark constraint definitions from ['/home/kasvegas/automlbenchmark/resources/constraints.yaml'].

Loading openml suite 411.



-------------------------------------------------------------

Starting job local.openml_s_411.test.doa_bwin.0.RandomForest.

Assigning 4 cores (total=10) for new task doa_bwin.

Assigning 8451 MB (total=15984 MB) for new doa_bwin task.

[MONITORING] [python3 [86101]] CPU Utilization: 0.0%

[MONITORING] [python3 [86101]] Memory Usage: 34.3%

[MONITORING] [python3 [86101]] Disk Usage: 56.5%

Running task doa_bwin on framework RandomForest with config:

TaskConfig({'framework': 'RandomForest', 'framework_params': {'n_estimators': 2000}, 'framework_version': '1.3.2', 'type': 'classification', 'name': 'doa_bwin', 'openml_task_id': 361983, 'test_server': False, 'fold': 0, 'metric': 'auc', 'metrics': ['auc', 'logloss', 'acc', 'balacc'], 'seed': 376815885, 'job_timeout_seconds': 1200, 'max_runtime_seconds': 600, 'cores': 4, 'max_mem_size_mb': 8451, 'min_vol_size_mb': -1, 'input_dir': '/home/kasvegas/.cache/openml/org/openml/www', 'output_dir': '/home/kasvegas/automlbenchmark/results/randomforest.openml_s_411.test.local.20231221T174026', 'output_predictions_file': '/home/kasvegas/automlbenchmark/results/randomforest.openml_s_411.test.local.20231221T174026/predictions/doa_bwin/0/predictions.csv', 'tag': None, 'command': 'runbenchmark.py randomforest openml/s/411', 'git_info': {'repo': 'https://github.com/openml/automlbenchmark.git', 'branch': 'HEAD', 'commit': '73823c55d8674761aab25520a75cbeea065ca124', 'tags': ['stable', 'v2.1.7'], 'status': ['## HEAD (no branch)', '?? doa-bwin-balanced.ipynb', '?? doa-bwin.ipynb', '?? resources/benchmarks/bwin.yaml', '?? upload_balanced']}, 'measure_inference_time': False, 'ext': {}, 'quantile_levels': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9], 'type_': 'binary', 'output_metadata_file': '/home/kasvegas/automlbenchmark/results/randomforest.openml_s_411.test.local.20231221T174026/predictions/doa_bwin/0/metadata.json'})

Running cmd `/home/kasvegas/automlbenchmark/frameworks/RandomForest/venv/bin/python -W ignore /home/kasvegas/automlbenchmark/frameworks/RandomForest/exec.py`

Traceback (most recent call last):



  File "/home/kasvegas/automlbenchmark/frameworks/RandomForest/exec.py", line 13, in <module>



    import psutil



ModuleNotFoundError: No module named 'psutil'







Command '/home/kasvegas/automlbenchmark/frameworks/RandomForest/venv/bin/python -W ignore /home/kasvegas/automlbenchmark/frameworks/RandomForest/exec.py' returned non-zero exit status 1.

Traceback (most recent call last):

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 578, in run

    meta_result = self.benchmark.framework_module.run(self._dataset, task_config)

  File "/home/kasvegas/automlbenchmark/frameworks/RandomForest/__init__.py", line 31, in run

    return run_in_venv(__file__, "exec.py",

  File "/home/kasvegas/automlbenchmark/frameworks/shared/caller.py", line 134, in run_in_venv

    output, err = run_cmd(cmd, *args,

  File "/home/kasvegas/automlbenchmark/amlb/utils/process.py", line 281, in run_cmd

    raise e

  File "/home/kasvegas/automlbenchmark/amlb/utils/process.py", line 255, in run_cmd

    completed = run_subprocess(str_cmd if params.shell else full_cmd,

  File "/home/kasvegas/automlbenchmark/amlb/utils/process.py", line 98, in run_subprocess

    raise subprocess.CalledProcessError(retcode, process.args, output=stdout, stderr=stderr)

subprocess.CalledProcessError: Command '/home/kasvegas/automlbenchmark/frameworks/RandomForest/venv/bin/python -W ignore /home/kasvegas/automlbenchmark/frameworks/RandomForest/exec.py' returned non-zero exit status 1.

Loading metadata from `/home/kasvegas/automlbenchmark/results/randomforest.openml_s_411.test.local.20231221T174026/predictions/doa_bwin/0/metadata.json`.

Metric scores: { 'acc': nan,

  'app_version': '2.1.7',

  'auc': nan,

  'balacc': nan,

  'constraint': 'test',

  'duration': nan,

  'fold': 0,

  'framework': 'RandomForest',

  'id': 'openml.org/t/361983',

  'info': 'CalledProcessError: Command '

          "'/home/kasvegas/automlbenchmark/frameworks/RandomForest/venv/bin/python "

          '-W ignore '

          "/home/kasvegas/automlbenchmark/frameworks/RandomForest/exec.py' "

          'returned non-zero exit st…',

  'logloss': nan,

  'metric': 'auc',

  'mode': 'local',

  'models_count': nan,

  'params': "{'n_estimators': 2000}",

  'predict_duration': nan,

  'result': nan,

  'seed': 376815885,

  'task': 'doa_bwin',

  'training_duration': nan,

  'type': 'binary',

  'utc': '2023-12-21T17:40:30',

  'version': '1.3.2'}

Job `local.openml_s_411.test.doa_bwin.0.RandomForest` executed in 0.430 seconds.

Scores saved to `/home/kasvegas/automlbenchmark/results/randomforest.openml_s_411.test.local.20231221T174026/scores/results.csv`.

Job `local.openml_s_411.test.doa_bwin.0.RandomForest` did not stop gracefully: Job `local.openml_s_411.test.doa_bwin.0.RandomForest` was interrupted.

Traceback (most recent call last):

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 226, in start

    self._run()

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 328, in _run

    self._add_result(result)

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 280, in _add_result

    self._on_new_result(result)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 339, in _process_results

    self._save(board)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 344, in _save

    self._save_global(board)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 354, in _save_global

    global_board.load().append(board).save()

  File "/home/kasvegas/automlbenchmark/amlb/results.py", line 203, in append

    scores = self.as_data_frame().append(to_append)

  File "/home/kasvegas/.local/lib/python3.10/site-packages/pandas/core/generic.py", line 6204, in __getattr__

    return object.__getattribute__(self, name)

AttributeError: 'DataFrame' object has no attribute 'append'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 147, in stop

    self._cancel()

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 184, in _cancel

    raise_in_thread(self.thread_id, CancelledError(f"Job `{self.name}` was interrupted."))

  File "/home/kasvegas/automlbenchmark/amlb/utils/process.py", line 437, in raise_in_thread

    ret = ctypes.pythonapi.PyThreadState_SetAsyncExc(tid, exc_class)

amlb.utils.process.CancelledError: Job `local.openml_s_411.test.doa_bwin.0.RandomForest` was interrupted.

All jobs executed in 2.720 seconds.

[MONITORING] [python3 [86101]] CPU Utilization: 8.5%

[MONITORING] [python3 [86101]] Memory Usage: 34.3%

[MONITORING] [python3 [86101]] Disk Usage: 56.5%

'DataFrame' object has no attribute 'append'

Traceback (most recent call last):

  File "/home/kasvegas/automlbenchmark/runbenchmark.py", line 189, in <module>

    res = bench.run(args.task, args.fold)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 211, in run

    results = self._run_jobs(jobs)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 253, in _run_jobs

    self.job_runner.start()

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 226, in start

    self._run()

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 328, in _run

    self._add_result(result)

  File "/home/kasvegas/automlbenchmark/amlb/job.py", line 280, in _add_result

    self._on_new_result(result)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 339, in _process_results

    self._save(board)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 344, in _save

    self._save_global(board)

  File "/home/kasvegas/automlbenchmark/amlb/benchmark.py", line 354, in _save_global

    global_board.load().append(board).save()

  File "/home/kasvegas/automlbenchmark/amlb/results.py", line 203, in append

    scores = self.as_data_frame().append(to_append)

  File "/home/kasvegas/.local/lib/python3.10/site-packages/pandas/core/generic.py", line 6204, in __getattr__

    return object.__getattribute__(self, name)

AttributeError: 'DataFrame' object has no attribute 'append'

[PGijsbers]

Thanks! It does work for me (on a fresh install), so let's see if we can figure out why it doesn't work on yours. I think the two errors have two different causes.

It looks like psutil wasn't installed in your RandomForest environment, even though it should have been. Please clean up the randomforest environment by removing the venv and .setup directories in frameworks/RandomForest, and then run the command again in setup mode by adding --setup=only to your command. The last line of output should read:

Setup of framework RandomForest completed successfully.

If that is not the case, please post the results here. If that is the case, you can probably just run the original command and it should work.

I suspect the AttributeError has a different cause. The DataFrame.append method got removed in Pandas 2.0, so it is likely you (accidentally) installed a newer pandas into the automl benchmark virtual environment (possibly while working on your notebooks). It might be best to recreate the virtual environment from which you run the automlbenchmark, and to keep that environment separate from the one(s) in which you do your analysis.

[kasvegas]

Hello Pieter,

Thank you once again for your prompt response. Your recommendations appear to have worked. You were correct, my pandas was >2.0. And after cleaning up the rf environment i successfully executed the rf benchmark and have a full set of results :)

I am in the process of trying the other frameworks now (hopefully I don't run into any more issues).

Thank you once again. I really appreciate what you and your colleagues have delivered here. I feel that my research using your tool is going to make some significant contributions to the field of gambling studies.

[kasvegas]

@PGijsbers. Thanks again for your help. I successfully ran 8 of the frameworks. However, I am running into issues with autosklearn, flaml, gama, and lighautoml. Here are the errors I am receiving:

For autosklearn: ModuleNotFoundError: No module named 'autosklearn'

For flaml: ModuleNotFoundError: NoResultError: best_iteration is only defined when early stopping is used.

For gama: ImportError: cannot import name 'SCORERS' from 'sklearn.metrics._scorer' (/home/kasvegas/automlbenchmark/frameworks/GAMA/venv/lib/python3.10/site-packages/sklearn/metrics/_scorer.py)

For lightautoml:
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
TypeError: cannot set 'get_record_history_wrapper' attribute of immutable type 'object'

Thank you in advance for your help, I appreciate that you are busy.

Sincerely,
Kas

[PGijsbers]

The look like installation errors to me. Have you followed the above steps to ensure it is reproducible?
If so, can you provide the output for the different installations (-setup=only)? autosklearn seemed to fail to install altogether, flaml likely failed to install a dependency, and gama likely installed an incompatible more recent version of scikit-learn. I am not sure why lightautoml required those permissions. Anyway, they may or may not each have separate reasons that the installation was unsuccessful, so for each log you can provide me I might be able to help you further. Please remember to first clean up the framework virtual environments as described above.

[kasvegas]

autosklearn setup.txt
flaml setup.txt
gama setup.txt
lightautoml setup.txt

Thank you for the prompt response. I have attached the logs for each framework set up. I executed the commands after cleaning up the virtual environments.

Sincerely,
Kas

[PGijsbers]

Making some suggestions for hotfixes here, unfortunately couldn't try them myself:

autosklearn: it looks like you are running Python 3.10, but auto-sklearn needs an old version of scikit-learn which only has prebuilt wheels until Python 3.9. AMLB officially only runs on 3.9, depending on framework dependencies you might need additional build tools in order to work with newer versions (as they need to build wheels, potentially compiling C).

flaml: it's likely one of the dependencies has been updated the integration script is no longer compatible. Might be able to deduce which one it is if I have the full trace of the error you reported earlier. You can also try fix upper limits to dependencies for frameworks by adding it to a requirements.txt in the respective framework folder.

gama: it looks like a newer version of scikit-learn is installed which is likely not compatible with GAMA. Try fixing the dependencies by adding scikit-learn<1.2 to requirements.txt in the GAMA framework folder and installing again.

lightautoml: I have no clue. @dev-rinchin do you recognize this error?

[kasvegas]

Thank you for the assistance.

I fixed the GAMA dependency like you suggested and it ran successfully...however not all folds completed. Is this a common scenario? I got "NoResultError: Ran out of input" for three folds and "NoResultError: Python int too large to convert to C ssize_t" for one another.

I re-tired autosklearn ensuring python3.9...it appears to be working now. I am currently running that one.

[PGijsbers]

GAMA does have a ~10% failure rate on the benchmark as a whole, but those errors tend to be biased towards particular datasets (like large datasets). Is this with openml/s/411? I don't see a reason why GAMA should have difficulties with those datasets, I'd have to look into it.

[kasvegas]

Thanks again. I ran GAMA again and it ran successfully on all folds.
Appreciate all your help on this.