filter alters input values in arrays

[jtackm]

Under julia 1.3, filter produces wrong output in the following example (see the tail of vals_filt):

n = 12000000
k = 257000000
vals = vcat(fill(0.5, n), fill(NaN, k))
vals_filt = filter(x -> x[2] < 1.0, collect(enumerate(vals)))

[Out]:
12000000-element Array{Tuple{Int64,Float64},1}:
 (1, 0.5) 
 (2, 0.5) 
 (3, 0.5) 
 (4, 0.5) 
 (5, 0.5) 
 (6, 0.5) 
 (7, 0.5) 
 (8, 0.5) 
 (9, 0.5) 
 (10, 0.5)
 (11, 0.5)
 (12, 0.5)
 (13, 0.5)
 ⋮        
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 
 (0, 0.0) 

In contrast, the equivalent

vals_filt_lc = [x for x in collect(enumerate(vals)) if x[2] < 1.0]

[Out]:
12000000-element Array{Tuple{Int64,Float64},1}:
 (1, 0.5)       
 (2, 0.5)       
 (3, 0.5)       
 (4, 0.5)       
 (5, 0.5)       
 (6, 0.5)       
 (7, 0.5)       
 (8, 0.5)       
 (9, 0.5)       
 (10, 0.5)      
 (11, 0.5)      
 (12, 0.5)      
 (13, 0.5)      
 ⋮              
 (11999989, 0.5)
 (11999990, 0.5)
 (11999991, 0.5)
 (11999992, 0.5)
 (11999993, 0.5)
 (11999994, 0.5)
 (11999995, 0.5)
 (11999996, 0.5)
 (11999997, 0.5)
 (11999998, 0.5)
 (11999999, 0.5)
 (12000000, 0.5)

works correctly.

Versioninfo (path information manually removed):

Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E7- 4870  @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, westmere)
Environment:
  JULIA_PKGDIR = <removed>
  JULIA_WORKER_TIMEOUT = 3600
  JULIA_HOME = <removed>
  JULIA_NUM_THREADS = 40

If I do any of the following, the bug disappears:

• replace the NaNs with regular floats
• reduce the input array size, e.g. by setting n to 11000000 and/or k to 256000000
• run the code on julia 1.2

[KristofferC]

Something with #31929 maybe?

[KristofferC]

It has something to do with the sizehint! at

julia/base/array.jl

Line 2367 in c7e9d9f

sizehint!(b, length(b))

Before the sizehint the 6 last elements are

6-element Array{Tuple{Int64,Float64},1}:
 (11999995, 0.5)
 (11999996, 0.5)
 (11999997, 0.5)
 (11999998, 0.5)
 (11999999, 0.5)
 (12000000, 0.5)

after

6-element Array{Tuple{Int64,Float64},1}:
 (0, 0.0)
 (0, 0.0)
 (0, 0.0)
 (0, 0.0)
 (0, 0.0)
 (0, 0.0)

Could it be something with #26201?

[KristofferC]

The NaN is a red herring

vals = vcat(fill(0.5, n), fill(1.5, k))

has the same bug

[jtackm]

I see, in my testing I replaced NaNs with a value that satisfied the predicate. Thanks for having a look!

[KristofferC]

Reverting #26201 does fix it. (which perhaps isn't so strange since it basically turns it into a noop for this case)