Numpy (and likely scipy) test failures ignored

[Flamefire]

The numpy EasyBlock runs the unittests as "cd .. && %(python)s -c 'import numpy; numpy.test(verbose=2)'"

However the numpy.test function returns True/False depending on success. As that is never checked the exit code is always 0 (unless a segfault or similar occurred). See https://github.com/numpy/numpy/blob/360ba0572483457837992d711a0a00580741fc88/numpy/_pytesttester.py#L213

This means we basically run the tests but ignore any failures other than hard crashes. And with 2019b numpy I see errors in the test suite (caused by having LDFLAGS set)
I don't think this is intentional and hence should be changed.

I assume scipy has the same problem as it is similar.

[ocaisa]

PR open for this for quite a while: #1744

[Flamefire]

That approach is quite... strange. Why not simply check the result of numpy.test?

[ocaisa]

Don't remember now, but there is the mysterious comment #605 (comment)

[Flamefire]

As the scipy thing is still open maybe we keep this open to not loose track?

[jfgrimm]

I see scipy test failures in older easyconfigs.
easyblocks PR #2862 updates the way we run tests going forward (keeping the old behaviour as default), by running runtests.py instead of using scipy.test(verbose).
The full test suite passes for me (2022b), and will exit with a non-zero code in the case of any failures.

[boegel]

With the changes in #2862, we're now not ignoring the scipy test suite anymore for recent versions (>= 1.9.0), so closing this 🥳