[Bug]: AutoMLBenchmark fails

[nicl-nno]

Fails for actual FEDOT version in "light" setup of AMLB for tabular data

[israel-cj]

Hi, there
I have been running FEDOT with the AutoMLBenchmark using shorter time constraints 5min, 10min, 30min. I understand shorter time constraints can origin not be enough time to find a workable pipeline, but I have a few regular problems that might not be related:

openml.org/t/2073,yeast,FEDOT,5min8c_gp3,0.0,multiclass,,neg_logloss,local,0.7.3.2#3d49382cd65e181b5ba5ec8f5dd4f844c88e8b9b,,"dev [https://github.com/israel-cj/automlbenchmark.git, master, c45e62f]",2024-09-12T19:26:54,47.6,,,,423089977.0,NoResultError: Invalid fitness after objective evaluation. Skipping the graph: (/n_scaling;)/n_logit,,,,,,,,,,,,,
openml.org/t/189356,albert,FEDOT,5min8c_gp3,4.0,binary,,auc,local,0.7.3.2#3d49382cd65e181b5ba5ec8f5dd4f844c88e8b9b,,"dev [https://github.com/israel-cj/automlbenchmark.git, master, c45e62f]",2024-09-13T02:03:40,333.1,,,,491438603.0,"NoResultError: Initial pipeline fit was failed due to: Shape of passed values is (306172, 66), indices imply (306172, 67). Check pipeline structure and the correctness of the data",,,,,,,,,,,,,
openml.org/t/359984,helena,FEDOT,5min8c_gp3,0.0,multiclass,,neg_logloss,local,0.7.3.2#3d49382cd65e181b5ba5ec8f5dd4f844c88e8b9b,,"dev [https://github.com/israel-cj/automlbenchmark.git, master, c45e62f]",2024-09-13T19:46:18,135.0,,,,1758775492.0,NoResultError: Initial pipeline fit was failed due to: Nodes can not be saved: BLOB longer than INT_MAX bytes. Continue. Check pipeline structure and the correctness of the data,,,,,,,,,,,,,
openml.org/t/360114,Higgs,FEDOT,5min8c_gp3,0.0,binary,,auc,local,0.7.3.2#3d49382cd65e181b5ba5ec8f5dd4f844c88e8b9b,,"dev [https://github.com/israel-cj/automlbenchmark.git, master, c45e62f]",2024-09-16T11:04:07,940.6,,,,1396818537.0,NoResultError: Initial pipeline fit was failed due to: Nodes can not be saved: Error binding parameter 1 - probably unsupported type.. Continue. Check pipeline structure and the correctness of the da…,,,,,,,,,,,,,
openml.org/t/3945,KDDCup09_appetency,FEDOT,5min8c_gp3,0.0,binary,,auc,local,0.7.3.2#3d49382cd65e181b5ba5ec8f5dd4f844c88e8b9b,,"dev [https://github.com/israel-cj/automlbenchmark.git, master, c45e62f]",2024-09-12T19:33:17,317.3,,,,382353392.0,TypeError: 'NoneType' object is not iterable,,,,,,,,,,,,,
openml.org/t/233211,diamonds,FEDOT,30min8c_gp3,4.0,regression,,neg_rmse,local,0.7.3.2#3d49382cd65e181b5ba5ec8f5dd4f844c88e8b9b,,"dev [https://github.com/israel-cj/automlbenchmark.git, master, c45e62f]",2024-09-16T11:53:00,708.0,,,,101930023.0,NoResultError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing th…,,,,,,,,,,,,,

I get other errors, but if those two are fixed, the number of empty results will decrease considerably. For the record, I am running on a slurm system (partition genoa); such systems have originated some problems with other automl frameworks. It might be the same with FEDOT or maybe a problem with the interaction AMLB-FEDOT?

I really appreciate any input you can provide.
Thanks

[nicl-nno]

@israel-cj thank you for this report! We will investigate the problem.

@dmitryglhf can you pls check this issue. It looks like FEDOT version in AMLB should be updated.

[dmitryglhf]

• Benchmark works well except of logloss metric, that in FEDOT presented as neg_log_loss:

with update this metric in exec.py at automlbenchmark it works correctly:

• Results on small 1h8c run give an error only on the kc1 dataset:

full_table.csv

• I'll explore this issue further:

I understand shorter time constraints can origin not be enough time to find a workable pipeline, but I have a few regular problems that might not be related:

[israel-cj]

Thank you for looking into this problem.
I can share also my results with all the time constrains, so you can look on those and let me know.

amlb_all.csv

Related to use neg_log_loss instead of logloss is already considered from our side.
Thank you.

[dmitryglhf]

Issue with logloss metric solved in openml/automlbenchmark@cdc660d