Investigate scipy 1.11.2 error

[gcroci2]

If we use the latest release of scipy, 1.11.2 (in the CI it's now fixed at 1.10.0), tests fail:

============================================================================================================ short test summary info =============================================================================================================
FAILED tests/test_trainer.py::TestTrainer::test_dataset_equivalence_pretrained - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_fout - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_ginet - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_ginet_class - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_ginet_sigmoid - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_incompatible_pretrained_no_Net - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_incompatible_pretrained_no_test - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_naive - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_no_valid_full_train - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_no_valid_provided - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional
FAILED tests/test_trainer.py::TestTrainer::test_sgat - numpy.linalg.LinAlgError: 0-dimensional array given. Array must be at least two-dimensional

It seems related to a numpy operation called in deeprank2.utils.community_pooling.community_detection. In particular, result = mc.run_mcl(matrix) throws the error, which seems to be caused by the fact that matrix is a 0-dimensional array. That is not true, I checked it and matrix.ndim gives 2 as a result. I already tried to downgrade numpy (and networkx) to the versions we have in the actions, but it didn't solve the issue.

The issue comes from the new scipy release (1.11.2); if I uninstall scipy and reinstall the version which is installed in the actions (1.10.0) everything works. But it's important to understand whether it's a bug in our package that the older scipy wasn't detecting (should be since it's only a minor release update).

Could you help me here? @cbaakman

[cbaakman]

Looks like a type problem. The matrix is of type: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_array.html

I see the error can be overcome by calling matrix.toarray(), before giving it to the markov clustering algorithm.

Shall I make this change?

[gcroci2]

Looks like a type problem. The matrix is of type: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_array.html

I see the error can be overcome by calling matrix.toarray(), before giving it to the markov clustering algorithm.

Shall I make this change?

Great you found it already! Yes sure go for it and assign me for reviewing it :)