Boolean indexing crashes win32-prerelease on 32-bit Vista

[goszlanyi]

julia> versioninfo()
Julia Version 0.3.0-prerelease+1358
Commit cb26338* (2014-02-01 14:56 UTC)
Platform Info:
  System: Windows (i686-w64-mingw32)
  CPU: Intel(R) Core(TM)2 Duo CPU     T9300  @ 2.50GHz
  WORD_SIZE: 32
  BLAS: libopenblas (DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

julia> a=[1:5]
5-element Array{Int32,1}:
 1
 2
 3
 4
 5

Then calling a[a.>3] crashes the REPL on 32-bit Vista.
I have also tried this win32-prerelease on 64-bit Win7 and it works fine.

[ivarne]

Do you get correct result for a.>3?

julia> a.>3
5-element BitArray{1}:
 false
 false
 false
  true
  true

[goszlanyi]

Yes.

[ivarne]

Boolean indexing is defined in https://github.com/JuliaLang/julia/blob/master/base/array.jl#L315-332. Do you get any error message? Does i crash on any of these lines, when you paste them in one by one?

a = [1:5];
b = a.>3;
checkbounds(a, b);
n = sum(I);
out = similar(a, n);

I don't have a 32 bit system, so it is hard for me to test and I just try to ask some questions to try to narrow things down.

[goszlanyi]

Thanks Ivar.
checkbounds(a,b) passes, it is sum(b) that crashes.

(I have also updated the title of the issue.)

[ivarne]

This is a slow way to debug the issue, but I'll try a few more iterations if nobody else shows up that is actually able to reproduce the problem. I noticed that BitArray uses an array of Uint64 to store the single values as individual bits.
What is your output from the following

dump(b)
bits(b.chunks[1])
count_ones(b[1])

[goszlanyi]

Thank you for your debug instructions. Here is the output:

julia> dump(b)
BitArray{1}
  chunks: Array(Uint64,(1,)) [0x0000000000000018]
  len: Int32 5
  dims: #undef

julia> bits(b.chunks[1])
"0000000000000000000000000000000000000000000000000000000000011000"

julia> count_ones(b[1])
ERROR: no method count_ones(Bool)

[ivarne]

Sorry, I made a typo.

count_ones(b.chunks[1])

[goszlanyi]

julia> count_ones(b.chunks[1])
2

[ivarne]

Yes, that was my intention. This seems very strange, and I am kind of stuck. Can you give me some outputs of the @which macro, just to see that the same methods gets called at your computer as in mine?

@which a[b]
@which sum(b)
@which nnz(b)
@which count_ones(b.chunks[1])

[goszlanyi]

julia> @which a[b]
getindex(A::Array{T,1},I::AbstractArray{Bool,1}) at array.jl:329
julia> @which sum(b)
sum(B::BitArray{N}) at bitarray.jl:1769
julia> @which nnz(b)
nnz(B::BitArray{N}) at bitarray.jl:1605
julia> @which count_ones(b.chunks[1])
count_ones(x::Uint64) at int.jl:184

I have just noted that nnz(b) also crashes.

[ivarne]

This is the implementation of nnz from bitarray.jl:1605

function nnz(B::BitArray)
    n = 0
    Bc = B.chunks
    for i = 1:length(Bc)
        n += count_ones(Bc[i])
    end
    return n
end

If count_ones works the only part left is to test 1:length(Bc). What is the output from [1:length(b.chunks)]?

[goszlanyi]

It works, so the mystery remains ...

julia> length(b.chunks)
1

julia> [1:length(b.chunks)]
1-element Array{Int32,1}:
 1

[ivarne]

This is the point where I give up. Can you try to just paste these lines into the REPL?

n = 0
Bc = b.chunks
for i = 1:length(Bc)
    n += count_ones(Bc[i])
end

[goszlanyi]

It works, the result is n=2.
I also think that our debugging options are exhausted.
Thank you for all your persistence,

[ivarne]

Can you try this with the @inbounds macro also?

n = 0
Bc = b.chunks
@inbounds for i = 1:length(Bc)
    n += count_ones(Bc[i])
end

I have no idea why that would make a difference, but that is what I feel we have narrowed it down to.

[goszlanyi]

Unfortunately, @inbounds did not make a difference, no crash this way.

[ivarne]

OK, last attemt:

Define a new function with exactly the same definition and try it on b

function my_nnz(B::BitArray)
    n = 0
    Bc = B.chunks
    @inbounds for i = 1:length(Bc)
        n += count_ones(Bc[i])
    end
    return n
end
my_nnz(b)

[goszlanyi]

Done.
Unfortunately, my_nnz works, while nnz still crashes.

[ihnorton]

@carlobaldassi is this related to any of the recent bitarray refactoring?

[carlobaldassi]

@ihnorton : I can't think of any way in which nnz could have been affected. The only thing which I changed recently has been the indexing into BitArrays, which is not involved here. Plus, if a user-defined function which is identical to a library function has a different behaviour, it seems like the only explanation must be a deeper bug, or in any case something unrelated which corrupts the memory.

[vtjnash]

It seems like somehow sys.dll may have snuck back into the dist files. see also #5701

[staticfloat]

It would be really unfortunate if this was due to shoddy packaging on my end. I'm currently refactoring a lot of my buildscripts to make them more generalized and reusable. Once I get everything hunky-dory again, I'll try building a set and seeing if I can reproduce this issue on my local machine.

[Sh4rK]

I too had the problem in #5701, and by removing sys.dll, it's working now.

[goszlanyi]

I can confirm that removing sys.dll solves the current issue.
Thanks!

[staticfloat]

@vtjnash Would it be correct of me to just put an rm julia-$(JULIA_COMMIT)/$(private_libdir)/sys.$(SHLIB_EXT) into our dist target, so that all distributions automatically have sys.{dll,so,dylib} taken out?

[vtjnash]

yeah, that would be good for now

[staticfloat]

Alright. I'm pretty close to merging #5527, so I've put a commit into there to this effect.

[vtjnash]

that's the wrong issue number, but sounds good to me

[Sh4rK]

This might be a stupid question, but why is it there in the first place? Deleting it seems to be a bit hackish. Also, there's a sys0.dll too, is that something related?

[aviks]

@Sh4rK sys.dll is the pre-compiled standard library. The primary benefit of pre-compiling that is quick startup. Everything else works without it. Unfortunately, it needs to be build processor specific, and hence should not be in binary builds. There is a plan to fix this: #5459

[JeffBezanson]

Closing as dup of #5459.