Switch to MKL versions of MXNet #442
Conversation
| - pip install -r requirements.docs.txt | ||
| - pip install -r requirements/requirements.txt | ||
| - pip install -r requirements/requirements.dev.txt | ||
| - pip install -r requirements/requirements.docs.txt | ||
|
|
There was a problem hiding this comment.
Since the files are now in a directory, consider renaming them to {core,dev,docs}.txt.
|
it seems like apache/mxnet#8532 is still not fixed and we still see crashes in environments where numpy uses MKL (such as any conda/anaconda setup). I would actually vote for reverting to the non-mkl version by default and add a section to the tutorials on speedups with MKL with the explicit hint that one needs a numpy without MKL. What do you think? |
|
Yes, the new MKL-DNN backend has accelerated the sockeye a lot. And recently we add more optimizations for sockeye (GNMT, transformer, etc), such as mshadow/342, mxnet/11335 and others PR (topk, copy2copy, etc) are on the way :) For example, the CPU inference performance is 8X faster with MKL-DNN as below table.
Thus, it's the right direction to enable MKL-DNN backend eventually. Alternatively, how about enable MKL-DNN and explicitly document this known issue? |
|
thanks for those numbers, that's great to hear! |
|
Do we have a log from a specific test that's failing with a specific setup? I just re-ran the system tests with both PyPI and Conda NumPy ( >>> np.__config__.show()
mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/home/ec2-user/miniconda3/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/home/ec2-user/miniconda3/include']
blas_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/home/ec2-user/miniconda3/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/home/ec2-user/miniconda3/include']
blas_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/home/ec2-user/miniconda3/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/home/ec2-user/miniconda3/include']
lapack_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/home/ec2-user/miniconda3/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/home/ec2-user/miniconda3/include']
lapack_opt_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/home/ec2-user/miniconda3/lib']
define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
include_dirs = ['/home/ec2-user/miniconda3/include']
>>> help(mxnet)
Help on package mxnet:
...
FILE
/home/ec2-user/miniconda3/lib/python3.6/site-packages/mxnet_mkl-1.2.0-py3.6-linux-x86_64.egg/mxnet/__init__.pyFor the unit and integration tests, I am seeing a weird issue of |
|
I'm now getting the following for both PyPI and Conda NumPy and both Conda and system (Amazon Linux) Python: Narrowing it down to this line, I get: Using Any ideas? |
|
It looks like the issue may be with a call to # (batch_size, total_num_filters, seq_len, num_scores=1)
conv_concat = mx.sym.concat(*conv_outputs, dim=1)If we manually specify elements to concatenate, everything works for multiple copies of the same element, but crashes otherwise. If we replace the above, we get the following results: All tests pass: conv_concat = mx.sym.concat(conv_outputs[0], conv_outputs[0], dim=1)
total_num_filters = self.num_filters[0] + self.num_filters[0]All tests pass: conv_concat = mx.sym.concat(conv_outputs[1], conv_outputs[1], dim=1)
total_num_filters = self.num_filters[1] + self.num_filters[1]Crashes, test aborts: conv_concat = mx.sym.concat(conv_outputs[0], conv_outputs[1], dim=1)
total_num_filters = self.num_filters[0] + self.num_filters[1]Any ideas? |
|
@mjdenkowski Hi, I cannot reproduce your error on my CPU/GPU machines with conda environment. |
|
I will try to find a minimal example that reproduces the issue. |
|
I was indeed not able to reproduce this with |
|
@tdomhan I update numpy and this duplicated iomp5.so bug cannot be reproduced. |
|
This issue has been fixed in the latest MKL package 2018 update 3 and the new numpy 1.14 will work fine. So, is it possible to update the dependency of numpy in the README? @tdomhan @mjdenkowski Dependencies Python3 |
This update switches our requirements over to MKL-DNN versions of MXNet (mxnet-mkl, mxnet-cu91mkl, etc.).
Using MKL dramatically improves performance on CPUs. Starting with version 1.2.0, MXNet also makes significantly better use of the open source version, MKL-DNN. This brings performance improvements to the pip-installable MXNet versions that are on par with building MXNet from source with the full version of MKL. Newer MKL versions of MXNet are also more compatible with various setups, such as installing Python via Conda, allowing us to more easily use them with Sockeye.
Pull Request Checklist
until you can check this box.
pytest)pytest test/system)./style-check.sh)sockeye/__init__.py. Major version bump if this is a backwards incompatible change.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.