Anaconda's scipy.test() fails with "OMP: Error #100: Fatal system error detected

[evfro]

• A brief description

SciPy is a python package for scientific computing. Normally it's installed with Anaconda distribution, that also ships Intel's MKL library (with LAPACK and BLAS routines).

• Expected results

Executing scipy.test() from within python completes with the report on all errors and failures discovered.

• Actual results (with terminal output if applicable)

scipy.test() doesn't complete and terminates python with the following error:

OMP: Error #100: Fatal system error detected.
OMP: System error #22: Invalid argument
Aborted (core dumped)

• Your Windows build number

Microsoft Windows [Version 10.0.14393]

• Steps / commands required to reproduce the error

Run from terminal:

python -c 'import scipy; scipy.test()'

• Strace of the failing command

https://gist.github.com/Evfro/2d0d83745a92d3f6e2f52ce3675a273d

The last thing before error there is the following line:
sched_getaffinity(0, 4, 0x7fc7757a7e00) = -1 EINVAL (Invalid argument)

• Required packages and commands to install

Install latest miniconda (Python 2.7):

wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh
bash Miniconda*.sh

Update apt-get (prevents some errors):

sudo apt-get update -qq

Install scipy, numpy and nose (for running tests):

conda install numpy scipy nose

This will additionally install MKL and libgfortran-3.
Note: test doesn't fail if nomkl versions of numpy and scipy are installed, however, MKL (especially its LAPACK and BLAS) is required for some other packages, so it's not really an option.

[deto]

Which packages is MKL required for? As far as I know, it was only added recently to their free-tier Linux installer so most things should work fine with nomkl.

[evfro]

For example, this ones fails at compilation time with exactly the same error (and it's not possible to install w/o MKL):
https://github.com/oseledets/ttpy
I guess, there could be many more scientific computing projects, that also rely on MKL's BLAS and LAPACK.

[evfro]

Sorry, my last comment was wrong. TTpy packages doesn't fail at compile time, but it fails when some functions are called (for example, tt.tensor).

What I've found is that setting environmental variable KMP_AFFINITY=disabled resolves the problem. However, I have no idea what implications it might have and not sure whether it's a proper workaround.

And it actually means, that the problem is not with MKL itself, but with OpenMP becoming faulty with any other value of KMP_AFFINITY (I tried all).

[srinath29]

Yeah.. And all the packages built on top of it. I was using statsmodels.api.GLSAR and it was also failing with the same problem. I opened a bug in statsmodels
statsmodels/statsmodels#3142

[srinath29]

Any solutions ? I am little anxious as one of my work depends on it.

[deto]

If nobody has a fix and you really need it, you might want to just run
Ubuntu in a VM. I've installed Anaconda with MKL doing this before. Not
as optimal as Bash on Windows but it'll get you moving.

You can also run much of the data science stack in Windows (not Bash) just
fine using Anaconda.

On Aug 7, 2016 12:29 AM, "srinath" notifications@github.com wrote:

Any solutions ? I am little anxious as one of my work depends on it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx0us8t2TFzKCfsjh6LgOmZUWQCX5ks5qdYlSgaJpZM4Jd6On
.

[srinath29]

Thanks for the suggestion. I am already doing that. Actually I am in final stages of development of a benchmark https://github.com/srinath29/UltimateAnalyticsBenchmark
I want to test it on Bash on windows and I have score of running it natively on windows also. So I am bit curious.

[evfro]

Have you tried setting KMP_AFFINITY=disabled? Does it really kills performance? I haven't tested yet, would be interesting to see.

[srinath29]

@evfro
Hey... It is working!!!!!
These are the results

See the score at the bottom. Bash is performing better than windows. Now I am not sure it is because making it disabled or it's bash's power. Can you do the same test on your system? I am very curious to see how it is working.
I think this should be labeled as workaround .

[evfro]

@srinath29
I haven't closely looked into your code, but the thing to keep in mind when measuring time is that some python time-measuring functions behave differently on Linux and Windows. The most reliable way I found is to use

from timeit import default_timer

Now, that said, I can confirm that there are performance differences in certain types of operations.
For example, this simple code

a = numpy.random.random((10000, 10000))
%timeit a.mean()

gives me on average 52ms on bash and 133ms on windows.

However, more involved calculations like computing sparse SVD (scipy.sparse.linalg.svds) give me roughly equal results: 151ms on bash against 156ms on windows (for 6040 by 3706 matrix with 99% sparsity).

And while this KMP_AFFINITY thing does the trick, I would really like to hear from core developers team of BashOnWindows whether this is an appropriate modification (and whether it can be made default without additional user actions). From Intel's website: "Thread affinity is supported on Windows systems and versions of Linux systems that have kernel support for thread affinity..." So the problem can be either due to a bug or due to a kernel-level limitation.

[davidghiurco]

does anyone know if there's a fix other than disabling MKL affinity? I'm using a numpy/scipy stack for one of my classes and i develop on Bash for windows and am wondering if there's any fix for this or if I still have to do it the hacky way (and can rest assured that unless professor runs BashOnWindows too, he won't get this error)

[drhirsch]

suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04

[davidghiurco]

how to get this build for Bash for Windows? or is it installed automatically with Windows update?

[drhirsch]

You need Windows 14936 or newer, and no it will not autoupgrade, even when you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder) so backup anything you might need.

[deto]

Can't you also run do-release-upgrade? So you don't lose all your files?

On Nov 19, 2016 2:42 PM, "Michael Hirsch" notifications@github.com wrote:

You need Windows 14936 or newer, and no it will not autoupgrade, even when
you get the latest build.
To upgrade WSL from Ubuntu 14.04 to Ubuntu 16.04, from Windows Command
Prompt

lxrun /uninstall
lxrun /install

That erases everything but your home directory (in the Ubuntu lxss folder)
so backup anything you might need.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#785 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAhrx8YV96Cxh8PsNLNj8_GrV5e1QiZqks5q_3tUgaJpZM4Jd6On
.

[drhirsch]

Some people do that, I wanted a clean start.

[bklooste]

Same with CNTK ., KMP_AFFINITY=disabled made it work

[stehufntdev]

Closing this per the comment above, please let us know if there are any regressions in this scenario:

"suggest CLOSE this issue as it was resolved with Build 14936 and installing new default Ubuntu 16.04"

[grayskripko]

@scienceopen it is possible to upgrade Ubuntu 14 to 16
According to https://media.readthedocs.org/pdf/wsl-guide/latest/wsl-guide.pdf

You can also upgrade to the latest version of Ubuntu with the following command (caution, this will take quite some time)!:
$ sudo -S apt-mark hold procps strace sudo
$ sudo -S env RELEASE_UPGRADER_NO_SCREEN=1 do-release-upgrade

I've got 16.04 on my machine after these steps