Segfault when testing on red hat 5, no useful output

[cournape]

Hi,

While looking at bcolz 0.8.0, I got some segfaults when running the test suite:

python -c "import bcolz; bcolz.test(verbose=True, heavy=True)"
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
bcolz version:     0.8.0
NumPy version:     1.8.1
Blosc version:     1.5.2 ($Date:: 2014-12-30 #$)
Blosc compressors: ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib']
Numexpr version:   not available (version >= 1.4.1 not detected)
Python version:    2.7.3 |Master 2.0.0.dev1-8b8019c (32-bit)| (default, Nov  8 2013, 08:53:43) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)]
Platform:          linux2-i686
Byte-ordering:     little
Detected cores:    4
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Performing the complete test suite!
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
............................('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.....('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
.('Checking compressors:', ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib'])
................................................................................................................................................................ssssss..............................Segmentation fault

I would have expected more output for verbose, if only to track down which test is failing. The test suite sometimes succeed, though.

[esc]

Thanks very much testing this! I have a couple of thoughts:

1. Could you try also the 0.7.3 release? That would help in determining if this is related to the recent refactor (cython definitions file)

2. I know the handcrafted test-suite isn't that ideal. You could try using nosetests to run the tests:

nosetests bcolz/tests/test_*.py -sv

Using -sv to print the test being run and any output could give a clue which test may be involved.

3. Lastly, please provide the backtrace left by the segfault. I am sure you know how to get it, but just in case, here is how I do it (if you have a better way let me know):

gdb python core

And then get the backtrace with:

bt

That's all for now. I have been seeing segfaults in bloscpack recently also, so I am very much interested in drilling down into this, since I think these things might be related. Also I have experienced segfaults in bcolz when using a python build with --with-pydebug but only when running the tests from pip installs, not from a clone. I am still working on all of these.

I have some more ideas, but let's check the above first, that should give us a few more clues to work with.

[cournape]

Regarding 2), I can't reproduce the segfault if I use nosetests in bcolz/tests/test_*.

Regarding 1) I see a segfault for 0.7.3 as well, but again only if I run the test suite with the script:

import bcolz
bcolz.test(verbose=True,heavy=True)

and even then, it does not always segfaults. When it does not, it passes

3): does not look very useful I am afraid:

#0  0x002cf3f5 in list_dealloc (op=0xb7824a0c) at Objects/listobject.c:309
#1  0x00378c5c in pattern_dealloc (self=0x880cb60) at ./Modules/_sre.c:1861
#2  0x002d841c in PyDict_Clear (op=0xb7ae9a44) at Objects/dictobject.c:891
#3  0x002d847d in dict_clear (mp=0xb7ae9a44) at Objects/dictobject.c:1972
#4  0x00339398 in call_function (f=0x8793e2c, throwflag=0) at Python/ceval.c:4005
#5  PyEval_EvalFrameEx (f=0x8793e2c, throwflag=0) at Python/ceval.c:2666
#6  0x00338ac9 in call_function (f=0x8793b5c, throwflag=0) at Python/ceval.c:4107
#7  PyEval_EvalFrameEx (f=0x8793b5c, throwflag=0) at Python/ceval.c:2666
#8  0x00338ac9 in call_function (f=0x879388c, throwflag=0) at Python/ceval.c:4107
#9  PyEval_EvalFrameEx (f=0x879388c, throwflag=0) at Python/ceval.c:2666
#10 0x002be2e1 in gen_send_ex (gen=0xb7819a54, arg=0x0, exc=0) at Objects/genobject.c:84
#11 0x002ce809 in listextend (self=0xb78619ac, b=0xb7819a54) at Objects/listobject.c:872
#12 0x002cea58 in list_init (self=0xb78619ac, args=0xb787010c, kw=0x0) at Objects/listobject.c:2458
#13 0x002f57f5 in type_call (type=<value optimized out>, args=0xb787010c, kwds=0x0) at Objects/typeobject.c:735
#14 0x002a059c in PyObject_Call (func=0x3aff00, arg=0xb787010c, kw=0x0) at Objects/abstract.c:2529
#15 0x00333b99 in call_function (f=0x879372c, throwflag=0) at Python/ceval.c:4239
#16 PyEval_EvalFrameEx (f=0x879372c, throwflag=0) at Python/ceval.c:2666
#17 0x00338ac9 in call_function (f=0x87935cc, throwflag=0) at Python/ceval.c:4107
#18 PyEval_EvalFrameEx (f=0x87935cc, throwflag=0) at Python/ceval.c:2666
#19 0x0033a05c in PyEval_EvalCodeEx (co=0xb7997d58, globals=0xb78deacc, locals=0x0, args=0x8785044, argcount=1, kws=0x8785048, kwcount=0, defs=0x0, defcount=0, 
    closure=0x0) at Python/ceval.c:3253
#20 0x00337f3d in call_function (f=0x8784edc, throwflag=0) at Python/ceval.c:4117
#21 PyEval_EvalFrameEx (f=0x8784edc, throwflag=0) at Python/ceval.c:2666
#22 0x0033a05c in PyEval_EvalCodeEx (co=0xb7aeb608, globals=0xb7adfa44, locals=0x0, args=0xb7886738, argcount=2, kws=0xb7eda038, kwcount=0, defs=0xb7afedb8, defcount=1, 
    closure=0x0) at Python/ceval.c:3253
#23 0x002c926c in function_call (func=0xb7af6e2c, arg=0xb788672c, kw=0xb78299bc) at Objects/funcobject.c:526
#24 0x002a059c in PyObject_Call (func=0xb7af6e2c, arg=0xb788672c, kw=0xb78299bc) at Objects/abstract.c:2529
#25 0x003335d9 in ext_do_call (f=0x8784d74, throwflag=0) at Python/ceval.c:4334
#26 PyEval_EvalFrameEx (f=0x8784d74, throwflag=0) at Python/ceval.c:2705
#27 0x0033a05c in PyEval_EvalCodeEx (co=0xb7aeb698, globals=0xb7adfa44, locals=0x0, args=0xb7886518, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#28 0x002c918a in function_call (func=0xb7af6e9c, arg=0xb788650c, kw=0x0) at Objects/funcobject.c:526
#29 0x002a059c in PyObject_Call (func=0xb7af6e9c, arg=0xb788650c, kw=0x0) at Objects/abstract.c:2529
#30 0x002af575 in instancemethod_call (func=0xb798a824, arg=0xb788650c, kw=0x0) at Objects/classobject.c:2578
#31 0x002a059c in PyObject_Call (func=0xb798a824, arg=0xb7861c2c, kw=0x0) at Objects/abstract.c:2529
#32 0x002fa31c in slot_tp_call (self=0xb78a7f0c, args=0xb7861c2c, kwds=0x0) at Objects/typeobject.c:5403
#33 0x002a059c in PyObject_Call (func=0xb78a7f0c, arg=0xb7861c2c, kw=0x0) at Objects/abstract.c:2529
#34 0x00333b99 in call_function (f=0x8787d24, throwflag=0) at Python/ceval.c:4239
#35 PyEval_EvalFrameEx (f=0x8787d24, throwflag=0) at Python/ceval.c:2666
#36 0x0033a05c in PyEval_EvalCodeEx (co=0xb7a7f890, globals=0xb7ae92d4, locals=0x0, args=0xb78864d8, argcount=2, kws=0xb7eda038, kwcount=0, defs=0xb7a85af8, defcount=1, 
    closure=0x0) at Python/ceval.c:3253
#37 0x002c926c in function_call (func=0xb7a82bc4, arg=0xb78864cc, kw=0xb7829934) at Objects/funcobject.c:526
#38 0x002a059c in PyObject_Call (func=0xb7a82bc4, arg=0xb78864cc, kw=0xb7829934) at Objects/abstract.c:2529
#39 0x003335d9 in ext_do_call (f=0x87866c4, throwflag=0) at Python/ceval.c:4334
#40 PyEval_EvalFrameEx (f=0x87866c4, throwflag=0) at Python/ceval.c:2705
#41 0x0033a05c in PyEval_EvalCodeEx (co=0xb7a7f770, globals=0xb7ae92d4, locals=0x0, args=0xb7886c18, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#42 0x002c918a in function_call (func=0xb7a82c6c, arg=0xb7886c0c, kw=0x0) at Objects/funcobject.c:526
#43 0x002a059c in PyObject_Call (func=0xb7a82c6c, arg=0xb7886c0c, kw=0x0) at Objects/abstract.c:2529
#44 0x002af575 in instancemethod_call (func=0xb798a7fc, arg=0xb7886c0c, kw=0x0) at Objects/classobject.c:2578
#45 0x002a059c in PyObject_Call (func=0xb798a7fc, arg=0xb7861bec, kw=0x0) at Objects/abstract.c:2529
#46 0x002fa31c in slot_tp_call (self=0xb78a7ecc, args=0xb7861bec, kwds=0x0) at Objects/typeobject.c:5403
#47 0x002a059c in PyObject_Call (func=0xb78a7ecc, arg=0xb7861bec, kw=0x0) at Objects/abstract.c:2529
#48 0x00333b99 in call_function (f=0x878654c, throwflag=0) at Python/ceval.c:4239
#49 PyEval_EvalFrameEx (f=0x878654c, throwflag=0) at Python/ceval.c:2666
#50 0x0033a05c in PyEval_EvalCodeEx (co=0xb7a7f890, globals=0xb7ae92d4, locals=0x0, args=0xb781c198, argcount=2, kws=0xb7eda038, kwcount=0, defs=0xb7a85af8, defcount=1, 
    closure=0x0) at Python/ceval.c:3253
#51 0x002c926c in function_call (func=0xb7a82bc4, arg=0xb781c18c, kw=0xb78298ac) at Objects/funcobject.c:526
#52 0x002a059c in PyObject_Call (func=0xb7a82bc4, arg=0xb781c18c, kw=0xb78298ac) at Objects/abstract.c:2529
#53 0x003335d9 in ext_do_call (f=0x878aef4, throwflag=0) at Python/ceval.c:4334
#54 PyEval_EvalFrameEx (f=0x878aef4, throwflag=0) at Python/ceval.c:2705
#55 0x0033a05c in PyEval_EvalCodeEx (co=0xb7a7f770, globals=0xb7ae92d4, locals=0x0, args=0xb781c378, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#56 0x002c918a in function_call (func=0xb7a82c6c, arg=0xb781c36c, kw=0x0) at Objects/funcobject.c:526
#57 0x002a059c in PyObject_Call (func=0xb7a82c6c, arg=0xb781c36c, kw=0x0) at Objects/abstract.c:2529
#58 0x002af575 in instancemethod_call (func=0xb798a8ec, arg=0xb781c36c, kw=0x0) at Objects/classobject.c:2578
#59 0x002a059c in PyObject_Call (func=0xb798a8ec, arg=0xb7861bac, kw=0x0) at Objects/abstract.c:2529
#60 0x002fa31c in slot_tp_call (self=0xb78e72ac, args=0xb7861bac, kwds=0x0) at Objects/typeobject.c:5403
#61 0x002a059c in PyObject_Call (func=0xb78e72ac, arg=0xb7861bac, kw=0x0) at Objects/abstract.c:2529
#62 0x00333b99 in call_function (f=0x877ed1c, throwflag=0) at Python/ceval.c:4239
#63 PyEval_EvalFrameEx (f=0x877ed1c, throwflag=0) at Python/ceval.c:2666
#64 0x0033a05c in PyEval_EvalCodeEx (co=0xb7a7f890, globals=0xb7ae92d4, locals=0x0, args=0xb7821978, argcount=2, kws=0xb7eda038, kwcount=0, defs=0xb7a85af8, defcount=1, 
    closure=0x0) at Python/ceval.c:3253
#65 0x002c926c in function_call (func=0xb7a82bc4, arg=0xb782196c, kw=0xb7829824) at Objects/funcobject.c:526
#66 0x002a059c in PyObject_Call (func=0xb7a82bc4, arg=0xb782196c, kw=0xb7829824) at Objects/abstract.c:2529
#67 0x003335d9 in ext_do_call (f=0x877ebb4, throwflag=0) at Python/ceval.c:4334
#68 PyEval_EvalFrameEx (f=0x877ebb4, throwflag=0) at Python/ceval.c:2705
#69 0x0033a05c in PyEval_EvalCodeEx (co=0xb7a7f770, globals=0xb7ae92d4, locals=0x0, args=0xb7886bf8, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#70 0x002c918a in function_call (func=0xb7a82c6c, arg=0xb7886bec, kw=0x0) at Objects/funcobject.c:526
#71 0x002a059c in PyObject_Call (func=0xb7a82c6c, arg=0xb7886bec, kw=0x0) at Objects/abstract.c:2529
#72 0x002af575 in instancemethod_call (func=0xb798a84c, arg=0xb7886bec, kw=0x0) at Objects/classobject.c:2578
#73 0x002a059c in PyObject_Call (func=0xb798a84c, arg=0xb7e58e6c, kw=0x0) at Objects/abstract.c:2529
#74 0x002fa31c in slot_tp_call (self=0xb78e3ecc, args=0xb7e58e6c, kwds=0x0) at Objects/typeobject.c:5403
#75 0x002a059c in PyObject_Call (func=0xb78e3ecc, arg=0xb7e58e6c, kw=0x0) at Objects/abstract.c:2529
#76 0x00333b99 in call_function (f=0x8789ed4, throwflag=0) at Python/ceval.c:4239
#77 PyEval_EvalFrameEx (f=0x8789ed4, throwflag=0) at Python/ceval.c:2666
#78 0x00338ac9 in call_function (f=0x84409cc, throwflag=0) at Python/ceval.c:4107
#79 PyEval_EvalFrameEx (f=0x84409cc, throwflag=0) at Python/ceval.c:2666
#80 0x0033a05c in PyEval_EvalCodeEx (co=0xb7997bf0, globals=0xb78dea44, locals=0x0, args=0x8420470, argcount=0, kws=0x8420470, kwcount=2, defs=0xb78e3d38, defcount=2, 
    closure=0x0) at Python/ceval.c:3253
#81 0x00337f3d in call_function (f=0x8420334, throwflag=0) at Python/ceval.c:4117
#82 PyEval_EvalFrameEx (f=0x8420334, throwflag=0) at Python/ceval.c:2666
#83 0x0033a05c in PyEval_EvalCodeEx (co=0xb7e61b18, globals=0xb7f0a35c, locals=0xb7f0a35c, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
    at Python/ceval.c:3253
#84 0x0033a0f3 in PyEval_EvalCode (co=0xb7e61b18, globals=0xb7f0a35c, locals=0xb7f0a35c) at Python/ceval.c:667
#85 0x00353376 in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=0xb7f0a35c, locals=0xb7f0a35c, flags=0xbfdb7968, arena=0x841f7a8)
    at Python/pythonrun.c:1353
#86 0x0035342e in PyRun_FileExFlags (fp=0x84139f8, filename=0xbfdc2ac5 "yo.py", start=257, globals=0xb7f0a35c, locals=0xb7f0a35c, closeit=1, flags=0xbfdb7968)
    at Python/pythonrun.c:1339
#87 0x00354b44 in PyRun_SimpleFileExFlags (fp=0x84139f8, filename=0xbfdc2ac5 "yo.py", closeit=1, flags=0xbfdb7968) at Python/pythonrun.c:943
#88 0x0035529a in PyRun_AnyFileExFlags (fp=0x84139f8, filename=0xbfdc2ac5 "yo.py", closeit=1, flags=0xbfdb7968) at Python/pythonrun.c:747
#89 0x00366881 in Py_Main (argc=1, argv=0xbfdc1aa4) at Modules/main.c:729
#90 0x08049023 in main (argc=2, argv=0xbfdc1aa4) at ./Modules/python.c:448
quit

[cournape]

I actually also see segfaults on windows 64 bits (bcolz 0.8.0, numpy 1.8.1, MKL, built with vs 2008 on python 2.7.3). It does not always segfaults, but when it does, it does at test_carray.large_viewDiskTest, when testing view and iterators.

[esc]

meh :/

[esc]

This is a long shot, but could you try running it from a git clone? Since I am not seeing the git info in the versions, I presume you are using a pip install or tarball?

[esc]

Can you check #121 and #122 and maybe use dynamic linking to test bcolz v0.8.0 with blosc v1.4.1?

Instructions for linking dynamically to blosc can be found here:

http://bcolz.blosc.org/install.html#installing-from-tarball-sources

[cournape]

@esc sorry for the late reply. I just went back to the issue, and can confirm that using an external libblosc (1.4.1 in canopy's case) fix the issue on every platform we support.

That may be irrelevant, but running the crashing bcolz on my mac VM worked flawlessly (the environment is exactly the same as we use automatic provisioning for our build environments at Enthought). This may suggest a CPU-specific issue, for example SSE/AVX support (my mac is the newest model, whereas my machine having the crash are older).

[esc]

@cournape thanks for the feedback, your findings are in-line with my gut feeling.

[FrancescAlted]

I have put some time on this, and I came up with a small code that can trigger a problem in valgrind:

import numpy as np
import bcolz

print("ENTERING")
N = int(1e5)

a = np.arange(N, dtype="uint64")
b = bcolz.carray(a)
c = b.view()
u = c.iter(3)
w = b.iter(2)
assert sum(a[3:]) == sum(u)
assert sum(a[2:]) == sum(w)
print("EXITING")

Part of the output in valgrind:

==7315== Invalid read of size 4
==7315==    at 0x4ECC3B3: PyObject_Free (obmalloc.c:994)
==7315==    by 0x4EAEB0A: func_dealloc (funcobject.c:459)
==7315==    by 0x4EC2046: insertdict_by_entry (dictobject.c:519)
==7315==    by 0x4EC549B: insertdict (dictobject.c:556)
==7315==    by 0x4EC549B: dict_set_item_by_hash_or_entry (dictobject.c:765)
==7315==    by 0x4EC549B: PyDict_SetItem (dictobject.c:818)
==7315==    by 0x4EC876C: _PyModule_Clear (moduleobject.c:139)
==7315==    by 0x4F42EE0: PyImport_Cleanup (import.c:508)
==7315==    by 0x4F4FCCA: Py_Finalize (pythonrun.c:459)
==7315==    by 0x4F65F74: Py_Main (main.c:670)
==7315==    by 0x585FEC4: (below main) (libc-start.c:287)
==7315==  Address 0x5d82020 is 2,784 bytes inside a block of size 3,336 free'd
==7315==    at 0x4C2BE10: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7315==    by 0x1C5F7F25: my_free (blosc.c:217)
==7315==    by 0x1C5F7F25: blosc_destroy (blosc.c:1793)
==7315==    by 0x1C597988: __pyx_pf_5bcolz_10carray_ext_6_blosc_destroy (carray_ext.c:2565)
==7315==    by 0x1C597988: __pyx_pw_5bcolz_10carray_ext_7_blosc_destroy (carray_ext.c:2546)
==7315==    by 0x4F2F22E: ext_do_call (ceval.c:4343)
==7315==    by 0x4F2F22E: PyEval_EvalFrameEx (ceval.c:2718)
==7315==    by 0x4F30C6D: PyEval_EvalCodeEx (ceval.c:3265)
==7315==    by 0x4EAE850: function_call (funcobject.c:526)
==7315==    by 0x4E7F322: PyObject_Call (abstract.c:2529)
==7315==    by 0x4F28822: PyEval_CallObjectWithKeywords (ceval.c:3902)
==7315==    by 0x4F4FC79: call_sys_exitfunc (pythonrun.c:1751)
==7315==    by 0x4F4FC79: Py_Finalize (pythonrun.c:424)
==7315==    by 0x4F65F74: Py_Main (main.c:670)
==7315==    by 0x585FEC4: (below main) (libc-start.c:287)
==7315== 

I have no completely nailed down the root of the double free, but I think it has to do with how views are released in carray objects, probably causing a double release of the referenced carray.

[FrancescAlted]

Hmm, I think that my previous note was a red herring. I am thinking now that the problem might be an optimization that I introduced in blosclz for blosc 1.5. @cournape could you please apply the patch below and tell me how it goes? Unfortunately I cannot reproduce the crash in my machine :-/

diff --git a/c-blosc/blosc/blosclz.c b/c-blosc/blosc/blosclz.c
index 01842b8..e9bbaf4 100644
--- a/c-blosc/blosc/blosclz.c
+++ b/c-blosc/blosc/blosclz.c
@@ -98,7 +98,7 @@
 #define BLOCK_COPY(op, ref, len, op_limit)    \
 { int ilen = len % CPYSIZE;                   \
   uint8_t *cpy = op + len;                    \
-  if (cpy + CPYSIZE - ilen <= op_limit) {     \
+  if (0) {                                    \
     FASTCOPY(op, ref, cpy);                   \
     ref -= (op-cpy); op = cpy;                \
   }                                           \
@@ -112,7 +112,7 @@
 }

 #define SAFE_COPY(op, ref, len, op_limit)     \
-if (abs(op-ref) < CPYSIZE) {                  \
+if (1) {                                      \
   for(; len; --len)                           \
     *op++ = *ref++;                           \
 }                                             \
@@ -120,7 +120,7 @@ else BLOCK_COPY(op, ref, len, op_limit);

 /* Copy optimized for GCC 4.8.  Seems like long copy loops are optimal. */
 #define GCC_SAFE_COPY(op, ref, len, op_limit) \
-if ((len > 32) || (abs(op-ref) < CPYSIZE)) {  \
+if (1) {                                      \
   for(; len; --len)                           \
     *op++ = *ref++;                           \
 }                                             \

[esc]

I am going to close this due to inactivity, feel free to re-open if the issue persists.