[Blocking issue] ValueError: need at least one array to concatenate

[Advitya17]

Hi EconML team, we're getting the below error in our builds:

Wondering if this logic should pass with empty arr in

EconML/econml/solutions/causal_analysis/_causal_analysis.py

Line 973 in 479362a

arr = np.concatenate([ensure_proper_dims(attr(inf))

and if a new version of econml needs to be released so that our repo's builds can get unblocked with that - https://github.com/microsoft/responsible-ai-toolbox?

Thank you very much for your support!

[kbattocchi]

This indicates that none of the models for your features were successfully trained when you were fitting the CausalAnalysis class. I don't think that this logic should succeed, but we could consider providing a more informative error message.

Looking at your logs, you can see that the models for the features are all failing to train, with the message

Exception caught when training model for feature Population: cannot set WRITEABLE flag to True of this array

which is the real issue that you're running into. It's not immediately clear to me whether this is an issue with EconML or with your code.

[Advitya17]

Thank you for the prompt response @kbattocchi . The error is starting from the CausalAnalysis call in our code https://github.com/microsoft/responsible-ai-toolbox/blob/d665107a9512a3934c7f2e9d44f58fd2b15f4fb6/responsibleai/responsibleai/managers/causal_manager.py#L407, and we're mostly dealing with pandas dataframes and using EconML's methods to fit our model. Looks like the code in

EconML/econml/solutions/causal_analysis/_causal_analysis.py

Line 973 in 479362a

arr = np.concatenate([ensure_proper_dims(attr(inf))

has several np operations; I wonder if any of those np calls may be returning unwriteable arrays.

Someone hit the same error when metainfo is changed - open-mmlab/mmdetection#3628

If there's a casting/metadata issue while invoking any of those np methods, I wonder if resolving the ValueError statement may pass our unit tests even if the model is untrained/spurious, or dk if no model prediction is a possible scenario with using CausalAnalysis.

[gaugup]

@kbattocchi, we haven't changed anything in our code for causal for sometime now. Any case there was change that was introduced in the release of 2/14 that is causing this error now?

[kbattocchi]

@gaugup @Advitya17 although it's possible that something has changed on our end I think it's pretty unlikely - CausalAnalysis is basically in maintenance mode and so we only make minor changes (e.g. to support sklearn API breaking changes), although changes to other components of the EconML library that CausalAnalysis depends on could theoretically cause a break, but again I'm not aware of any changes that should result in this behavior.

It seems more likely that you're picking up some mutually incompatible versions of other libraries - for instance see pandas-dev/pandas#24839 for an incompatibility between certain pandas and numpy versions that causes the underlying error.

Do you have a local repro? And if so, can you output the results of pip list?

[imatiach-msft]

seems it occurs only with scikit-learn==1.4.1.post1 which was just released in Feb

[imatiach-msft]

apparently the error above was an issue with scikit-learn and numpy which was conveniently fixed with 1.5.1, see:

scikit-learn/scikit-learn#29018

using this write flag was added and then subsequently removed in newer versions of scikit-learn, so if using latest scikit-learn 1.5.1 the issue should go away