strange failure on linux and mac

[cpcloud]

I'm unable to use bcolz with multiprocessing when my data is a particular size:

here's a minimal failing example

import shutil
import numpy as np
from math import ceil
import bcolz
from multiprocessing import Pool

n = 2 ** 14
rootdir = 'foo'

try:
    shutil.rmtree(rootdir)
except OSError:
    pass

ct = bcolz.carray(np.random.rand(n) * 1000, rootdir=rootdir)
cs = 1000

slices = [slice(cs * i, cs * (i + 1)) for i in range(int(ceil(n / cs)))]


class Sum(object):
    def __init__(self, data):
        self.data = data

    def __call__(self, index):
        return self.data[index].sum()


if __name__ == '__main__':
    p = Pool()
    caller = Sum(ct)
    result = p.map(caller, slices)
    print(result)

[cpcloud]

for some reason, the failure happens when n = 2 ** 14, but not when n = 2 ** 14 - 1 or n = 2 ** 20.

[cpcloud]

i would imagine that since ctables can be pickled that this wouldn't be an issue, and i've been able to use ctable with multiprocessing in the past

[cpcloud]

I'm also able to reproduce this on both linux and mac osx yosemite

mac environment:

$ conda list | grep -e 'numpy\|bcolz\|python'
bcolz                     0.7.3                np19py27_0
numpy                     1.9.1                    py27_0
python                    2.7.9                         1

linux environment:

$ conda list | grep -e 'numpy\|bcolz\|python'
bcolz                     0.8.0                np19py27_0
numpy                     1.9.1                    py27_0
python                    2.7.9                         1

[cpcloud]

i've found a working revision, so i'm going to bisect

[cpcloud]

looks like 89253f1 is the culprit

guessing that blosc introduced something

don't have the bandwidth to delve into blosc right now

[esc]

Check #121 also seems blosc v1.5.2 related.

[esc]

What is the failure btw? For me, it just hangs w/o doing anything at all.

[cpcloud]

Yep forgot to say what was actually happening. It hangs for me without doing anything.

[cpcloud]

Is there any way we can revert the version of blosc that introduced this bug?

[esc]

We are currently discussing action for this on the mailinglist(s):

If it's urgent you can always link bcolz against and older blosc, would suggest using 1.4.1:

http://bcolz.blosc.org/install.html#installing-from-tarball-sources

[FrancescAlted]

After thinking a bit about this, my hunch is that this problem has something to do with the new multithreading mode implemented in c-blosc in 1.5. To make this run with c-blosc 1.5 series, just add:

bcolz.blosc_set_nthreads(1)

at the beginning of your script.

Hmm, the new threading implementation in 1.5 is having more implications than anticipated :/

[FrancescAlted]

After holding the GIL during compression/decompression (PR #166) fixes this issue.

[mrocklin]

Hooray for the GIL! (part of me just died saying that ;))

[FrancescAlted]

In fact, Stefan Behnel (from the Cython team) always said that the GIL was actually a good thing. I think I understand him more now :)

[esc]

Does that mean, we can bump to Blosc 1.5.x again?