Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Sampling fails on mxnet==1.5.0 on Linux? #16135

Closed
lostella opened this issue Sep 10, 2019 · 8 comments
Closed

Sampling fails on mxnet==1.5.0 on Linux? #16135

lostella opened this issue Sep 10, 2019 · 8 comments

Comments

@lostella
Copy link
Contributor

lostella commented Sep 10, 2019

Description

Some random sampling functions seem not to be working on mxnet version 1.5.0 on Linux.

Environment info (Required)

----------Python Info----------
Version      : 3.7.4
Compiler     : GCC 8.3.0
Build        : ('default', 'Aug 14 2019 12:09:51')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 19.2.3
Directory    : /usr/local/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version      : 1.5.0
Directory    : /usr/local/lib/python3.7/site-packages/mxnet
Commit Hash   : 75a9e187d00a8b7ebc71412a02ed0e3ae489d91f
Library      : ['/usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so']
Build features:
✖ CUDA
✖ CUDNN
✖ NCCL
✖ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✔ CPU_SSE4_1
✔ CPU_SSE4_2
✖ CPU_SSE4A
✔ CPU_AVX
✖ CPU_AVX2
✖ OPENMP
✖ SSE
✔ F16C
✖ JEMALLOC
✖ BLAS_OPEN
✖ BLAS_ATLAS
✖ BLAS_MKL
✖ BLAS_APPLE
✔ LAPACK
✖ MKLDNN
✔ OPENCV
✖ CAFFE
✖ PROFILER
✔ DIST_KVSTORE
✖ CXX14
✖ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
----------System Info----------
Platform     : Linux-4.9.184-linuxkit-x86_64-with-debian-10.0
system       : Linux
node         : 393443f5954f
release      : 4.9.184-linuxkit
version      : #1 SMP Tue Jul 2 22:58:16 UTC 2019
----------Hardware Info----------
machine      : x86_64
processor    :
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       39 bits physical, 48 bits virtual
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               142
Model name:          Intel(R) Core(TM) i7-7660U CPU @ 2.50GHz
Stepping:            9
CPU MHz:             2500.000
BogoMIPS:            4992.00
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            4096K
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht pbe syscall nx pdpe1gb lm constant_tsc rep_good nopl xtopology nonstop_tsc pni pclmulqdq dtes64 ds_cpl ssse3 sdbg fma cx16 xtpr pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch kaiser fsgsbase bmi1 hle avx2 bmi2 erms rtm xsaveopt arat
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0416 sec, LOAD: 0.6791 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0579 sec, LOAD: 1.1694 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0602 sec, LOAD: 0.8697 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0628 sec, LOAD: 1.0556 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0343 sec, LOAD: 0.8659 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0030 sec, LOAD: 0.1486 sec.
----------Environment----------

I'm using Python, pip-freeze gives:

certifi==2019.6.16
chardet==3.0.4
graphviz==0.8.4
idna==2.8
mxnet==1.5.0
numpy==1.17.2
requests==2.22.0
urllib3==1.25.3

Error Message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 68, in sample_normal
  File "/usr/local/lib/python3.7/site-packages/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke
    ctypes.byref(out_stypes)))
  File "/usr/local/lib/python3.7/site-packages/mxnet/base.py", line 253, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: vector::_M_range_insert

Minimum reproducible example

import mxnet as mx
s = mx.nd.sample_normal(mu=mx.nd.array([10.0]), sigma=mx.nd.array([0.5]))

Edit: the problem seems to occur on vanilla mxnet==1.5.0 as well; edited the issue description above.

@mxnet-label-bot
Copy link
Contributor

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended label(s): Build

@lostella
Copy link
Contributor Author

lostella commented Sep 10, 2019

This did not occur in mxnet_mkl-1.4.1, and appears to be still occurring in mxnet_mkl-1.6.0b20190909

@lostella lostella changed the title Sampling fails on mxnet-mkl==1.5.0? Sampling fails on mxnet==1.5.0? Sep 10, 2019
@lostella lostella changed the title Sampling fails on mxnet==1.5.0? Sampling fails on mxnet==1.5.0 on Linux? Sep 10, 2019
@ChaiBapchya
Copy link
Contributor

@mxnet-label-bot add [Operator, Bug]

@ChaiBapchya
Copy link
Contributor

Thanks for pointing out.
I was able to reproduce this issue for pip install mxnet==1.5.0, pip install mxnet==1.6.0

No error for pip install mxnet==1.4.0
So likely introduced in patch release 1.4.1

@lostella
Copy link
Contributor Author

No error for pip install mxnet==1.4.0
So likely introduced in patch release 1.4.1

I was not able to reproduce on 1.4.1, so probably introduced in 1.5.0

@reminisce
Copy link
Contributor

Fixed in #16139.

@lostella
Copy link
Contributor Author

@reminisce does the fix also fix sampling other than sample_normal? All other sampling functions also raise the same error on 1.5.0:

  • s = mx.nd.sample_uniform(low=mx.nd.array([10.0]), high=mx.nd.array([11.5]))
  • s = mx.nd.sample_negative_binomial(k=mx.nd.array([3]), p=mx.nd.array([0.5]))
  • s = mx.nd.sample_poisson(lam=mx.nd.array([1.5]))
  • ...

@asmushetzel
Copy link
Contributor

asmushetzel commented Sep 11, 2019

The fix provided does fix all the samplers (i.e. for all distributions).

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants