Inconsistent results from `===` on non-power-of-2 primitives

[NHDaly]

Here is an MRE we tracked down from our real code:

julia> primitive type ByteString18 (18 * 8) end

julia> a = reinterpret(ByteString18, ntuple(_->0x0, 18))
ByteString18(0x000000000000000000000000000000000000)

julia> (a,) === (a,)
false

julia> (a,) === (a,)
true

julia> (a,) === (a,)
false

julia> (a,) === (a,)
false

julia> (a,) === (a,)
false

julia> (a,) === (a,)
true

This is definitely wrong.
This reproduces for me on ARM Mac, x86 Linux, 1.10, and 1.12.

And it's not unique to reinterpret, it shows up with zext as well:

julia> primitive type ByteString18 (18 * 8) end

julia> a = Core.zext_int(ByteString18, 0x0)
ByteString18(0x000000000000000000000000000000000000)

julia> (a,) === (a,)  # should obviously be `true`
false

And also, it's not unique to being all 0s, it works for reinterpreting any values in there.

julia> a = reinterpret(ByteString18, ntuple(_->0xaf, 18)); C = 10000; sum(@eval ((a,) === (a,)) for _ in 1:C) / C
0.7306

So far, we've only ever seen this during constant-folding, but I'm not sure if that's 100% true or not.

Finally, as you'd expect, this issue doesn't seem to affect primitive types that have standard power-of-2 integer sizes: 1,2,4,8,16,32. But most of the rest are affected:

ratio of times `(a,) === (a,)` returns `true` out of 3,000 iterations, for different primitive type byte-widths.

Code to generate the above plot:

const MAX_PRIMITIVE_BYTE_STRING = 16 * 8
for BYTES in 1:MAX_PRIMITIVE_BYTE_STRING
    name = Symbol("ByteString$(BYTES)")
    BITS = BYTES * 8
    @eval begin
        primitive type $(name) $(BITS) end
    end
end

out = let iters = 3000, out = Float64[]
    for N in 1:32
        T = eval(Symbol("ByteString$(N)"))
        a = reinterpret(T, ntuple(_->0x0, N))
        push!(out, sum(@eval(($a,) === ($a,)) for _ in 1:iters) / iters)
    end
    out
end

using Plots

plot(out)
vline!([1,2,4,8,16,32])

[vchuravy]

Well, I guess the warning was justified... #58262

[NHDaly]

Hah, damn. I never saw that line in the docstring documentation page. ......seems like quite a gotcha lol 😅

And if that's true, and we put that warning back, it should also be in the docstring. In fact there should be a big fat warning in the docstring about this, if it's not actually supported.

[oscardssmith]

This is definitely supposed to be supported. IMO we should fix the bug, and put the warning back for another couple years :)

[NHDaly]

Thanks Oscar, that would be great.
This is a pretty frustrating thing to have discovered, pretty late into a high-urgency project at work. 😢

We were really thrilled at how good LLVM's codegen was for operations on these primitive types, rather than trying to replicate them with structs or NTuples of UInt8s or whatever. And we've built a lot of infrastructure on this choice, only to find this crack in the foundation. It'd be amazing to find and fix this. 🙏 Thank you!

[Seelengrab]

Do you also see this when the primitive object is not a global? I can reproduce this, but only in some circumstances where a global is accessed:

julia> module Foo
           primitive type ByteString18 18*8 end
           a = reinterpret(ByteString18, ntuple(_ -> 0x0, 18))
           res = (a,) === (a,)
           f(x) = (x,) === (x,)
           res2 = f(a)
           g() = (a,) === (a,)
           res3 = g()
           function h()
               x = reinterpret(ByteString18, ntuple(_ -> 0x0, 18))
               return (x,) === (x,)
           end
           res4 = h()
       end; using .Foo; (Foo.res, Foo.res2, Foo.res3, Foo.res4)
(false, true, false, true)

The first & third result flicker between true & false, while the second and fourth are consistently true.

[NHDaly]

@Seelengrab that is what i've seen as well. I thought that this was a proxy for constant-folding. But I guess probably res2 = f(a) should also be able to constant-fold through the function? Not sure.
But yes, this matches what I have seen myself.

[oscardssmith]

I believe this is a bug in object initialization. When we create a primitive type X 18 the layout LLVM is expecting is a 32 byte object where the last 14 bytes are zeroed. We likely aren't zeroing the end bytes, and are thus seeing issues.

[NHDaly]

💡 aha, makes sense, oscar! That's very interesting.
Even though LLVM has support for i18 types, they still require you to lay out the memory with 32-bytes?

[Seelengrab]

I don't think that's it - LLVM shouldn't care, as long as whatever ends up doing the comparison doesn't read uninitialized memory. If this were a general problem with ===, g should also be nondeterministic, but it isn't. Sadly it seems pretty difficult to figure out what exactly is doing the comparison, and what specifically differs between g and h above, as even h has tons of dynamic dispatch inside.

[oscardssmith]

@Seelengrab The cases that are deterministic are those where LLVM constant folds the answer:

julia> @code_llvm f(a)
; Function Signature: f(Main.ByteString18)
;  @ REPL[4]:1 within `f`
define i8 @julia_f_3830(i144 zeroext %"x::ByteString18") local_unnamed_addr #0 {
top:
  ret i8 1
}
julia> @code_llvm h()
; Function Signature: h()
;  @ REPL[8]:1 within `h`
define i8 @julia_h_2924() local_unnamed_addr #0 {
top:
  %jlcallframe1 = alloca [2 x ptr], align 8
  %gcframe2 = alloca [3 x ptr], align 16
  call void @llvm.memset.p0.i64(ptr align 16 %gcframe2, i8 0, i64 24, i1 true)
  %thread_ptr = call ptr asm "movq %fs:0, $0", "=r"() #10
  %tls_ppgcstack = getelementptr inbounds i8, ptr %thread_ptr, i64 -8
  %tls_pgcstack = load ptr, ptr %tls_ppgcstack, align 8
  store i64 4, ptr %gcframe2, align 8
  %frame.prev = getelementptr inbounds nuw i8, ptr %gcframe2, i64 8
  %task.gcstack = load ptr, ptr %tls_pgcstack, align 8
  store ptr %task.gcstack, ptr %frame.prev, align 8
  store ptr %gcframe2, ptr %tls_pgcstack, align 8
;  @ REPL[8]:2 within `h`
; ┌ @ ntuple.jl:19 within `ntuple`
   %0 = call nonnull ptr @j__ntuple_2926(i64 signext 18)
; └
; ┌ @ essentials.jl:765 within `reinterpret`
   %.tag_addr = getelementptr inbounds i8, ptr %0, i64 -8
   %.tag = load atomic i64, ptr %.tag_addr unordered, align 8
   %1 = and i64 %.tag, -16
   %2 = inttoptr i64 %1 to ptr
   %gc_slot_addr_0 = getelementptr inbounds nuw i8, ptr %gcframe2, i64 16
   store ptr %0, ptr %gc_slot_addr_0, align 8
   %3 = call i8 @j_isprimitivetype_2927(ptr nonnull %2)
   %4 = trunc i8 %3 to i1
   br i1 %4, label %L5, label %guard_exit4

L5:                                               ; preds = %top
   store ptr null, ptr %gc_slot_addr_0, align 8
; │ @ essentials.jl:766 within `reinterpret`
   call void @ijl_error(ptr nonnull @"_j_str_bitcast: value not a prim...#1")
   unreachable

guard_exit4:                                      ; preds = %top
; │ @ essentials.jl:769 within `reinterpret`
   store ptr @"+Main.ByteString18#2929.jit", ptr %jlcallframe1, align 8
   %5 = getelementptr inbounds nuw i8, ptr %jlcallframe1, i64 8
   store ptr %0, ptr %5, align 8
   %6 = call nonnull ptr @ijl_apply_generic(ptr nonnull @"jl_global#2928.jit", ptr nonnull %jlcallframe1, i32 2)
   %frame.prev11 = load ptr, ptr %frame.prev, align 8
   store ptr %frame.prev11, ptr %tls_pgcstack, align 8
; └
;  @ REPL[8]:3 within `h`
  ret i8 1
}

[NHDaly]

Aha, so my Issue title is exactly opposite to the truth.
It's more like "Inconsistent results from runtime eval of === on non-power-of-2 primitives"?