"Tuple field type cannot be Union{}" error in all 1.10 and 1.11 versions

[aplavin]

I've been running some package tests on the upcoming julia version (1.10rc) and noticed this error newly introduced there, compared to 1.9:

julia> Tuple{Union{}}
# 1.9:
Tuple{Union{}}
# 1.10:
ERROR: Tuple field type cannot be Union{}

Not sure if introduced deliberately or as a side-effect of some other change, but it breaks perfectly working code. And I don't see it anywhere in Tuple docs that it shouldn't support all element types.

The above is an MWE, and below is the simplified situation where I actually encountered it:

julia> X = StructArray(a=Union{}[], b=[]);
# 1.10 throws error right here
# 1.9 lets you proceed just fine

# can work with the non-Union{} column:
julia> X.b
Any[]

# can retrieve the Union{} column as well, of course cannot access its elements
julia> X.a |> typeof
Vector{Union{}} (alias for Array{Union{}, 1})

[N5N3]

This is a deliberate change, see #49111.

[vtjnash]

You didn't include a stacktrace, but it looks like this calls collect_structarray, which calls Core.Compiler.return_type, which is a private API in a private package with no documentation or stability guarantees. The closest actual API is Base.promote_op() which should work for this and is documented as such (though using it does not quite accurately meet the AbstractArray or collect APIs--c.f. the implementation of those in Base and how it tries to avoid them).

https://github.com/JuliaArrays/StructArrays.jl/blob/99f05561cab19fb23f478a12a8429764871ccff3/src/collect.jl#L44

help?> Core.Compiler.return_type
  │ Warning
  │
  │  The following bindings may be internal; they may change or be removed in future versions:
  │
  │    •  Core.Compiler
  │
  │    •  Core.Compiler.return_type

  No documentation found for private symbol.

[aplavin]

The MWE doesn't depend on return_type at all. And I don't think the structarray example does either.

See:

julia> arrs = (Union{}[], Int[])

# works on 1.9 – broken on 1.10rc
julia> Tuple{map(eltype, arrs)...}
ERROR: Tuple field type cannot be Union{}

[vtjnash]

Not all apply_type expressions are meaningful. They can return errors

[aplavin]

Creating a Tuple type with arbitrary types inside seems to make total sense, that's one of the most fundamental Julia types. I haven't seen anywhere in the docs that Tuple only supports a subset of Julia types. And this limitation does sound strange, doesn't it?

[aplavin]

So, should this error be removed?
Both:

• this change looks unambiguously breaking, perfectly working code that doesn't rely on internals stops working
• the behavior itself is totally sensible, even aside from the first point; surely one can create a Tuple type with any element types, no exceptions are documented and Union{} as element type does work elsewhere.

[rdboyes]

Running into this error using density() from AlgebraOfGraphics as well - a simple MWE from here: MakieOrg/AlgebraOfGraphics.jl#472 (comment)

julia> draw(
           data((x=randn(100), y=randn(100))) *
           mapping(:x, :y) *
           AlgebraOfGraphics.density() *
           visual(Contour)
       )
ERROR: Tuple field type cannot be Union{}
Stacktrace:
  [1] map(f::Function, d::Dictionaries.Indices{Union{}})
    @ Dictionaries ~/.julia/packages/Dictionaries/7aBxp/src/map.jl:91
  [2] unnest(vs::Vector{@NamedTuple{}}, indices::Dictionaries.Indices{Union{}})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layer.jl:81
  [3] unnest_dictionaries(vs::Vector{@NamedTuple{}})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layer.jl:84
  [4] map(f::AlgebraOfGraphics.var"#193#194"{@NamedTuple{…}}, processedlayer::ProcessedLayer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layer.jl:101
  [5] (::AlgebraOfGraphics.DensityAnalysis{…})(input::ProcessedLayer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/transformations/density.jl:30
  [6] call_composed
    @ Base ./operators.jl:1045 [inlined]
  [7] call_composed
    @ Base ./operators.jl:1044 [inlined]
  [8] (::ComposedFunction{AlgebraOfGraphics.Visual, AlgebraOfGraphics.DensityAnalysis{…}})(x::ProcessedLayer)
    @ Base ./operators.jl:1041
  [9] process(layer::Layer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/processing.jl:102
 [10] iterate(g::Base.Generator, s::Vararg{Any})
    @ Base ./generator.jl:47 [inlined]
 [11] collect(itr::Base.Generator{Layers, typeof(AlgebraOfGraphics.process)})
    @ Base ./array.jl:834
 [12] map
    @ ./abstractarray.jl:3310 [inlined]
 [13] ProcessedLayers(a::Layer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:41
 [14] compute_axes_grid(d::Layer; axis::@NamedTuple{}, palettes::@NamedTuple{})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:114
 [15] compute_axes_grid
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:110 [inlined]
 [16] compute_axes_grid(fig::Figure, d::Layer; axis::@NamedTuple{}, palettes::@NamedTuple{})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:100
 [17] compute_axes_grid
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/algebra/layers.jl:97 [inlined]
 [18] #241
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:21 [inlined]
 [19] update
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:10 [inlined]
 [20] plot!(fig::Figure, d::Layer; axis::@NamedTuple{}, palettes::@NamedTuple{})
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:21
 [21] plot!
    @ ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:16 [inlined]
 [22] (::AlgebraOfGraphics.var"#245#246"{…})(f::Figure)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:48
 [23] update
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:10 [inlined]
 [24] #draw#244
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:47 [inlined]
 [25] draw(d::Layer)
    @ AlgebraOfGraphics ~/.julia/packages/AlgebraOfGraphics/tbMEb/src/draw.jl:44
 [26] top-level scope
    @ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types.

[vtjnash]

As with the previous person, this is due to the Dictionaries package using the disallowed private symbol Core.Compiler.return_type. Open an issue there and link #52385 (comment)?

[aplavin]

This issue (the one discussed here, don't know about AoG) is independent on return_type() though, but due to a breaking change in Julia 1.10. Even more, it's breaking completely sensible behavior of being able to represent Tuple{T} for T = Union{}.

There's nothing fundamentally bad with breaking changes, they should just be clearly communicated. For now, Julia promises no breaking changes in 1.x, which this clearly contradicts.

[nsajko]

The empty union, Union{}, is the type with no instances, thus Tuple{Union{}}, and any other tuple type with an empty union field, is also the type with no instances, so I guess we should have Tuple{Union{}} == Union{} hold. I don't see how it makes sense for Tuple{Union{}} to error.

I admit this might be difficult to fix, though.

[vtjnash]

Having no instances is not the same as having no subtypes. Those are 2 orthogonal properties.

We could try to return Union{} here, as that was the first attempt before making it an error, but much code behaved poorly with that, so it would be a breaking change. Making it an error--for this case which was already buggy--was not a breaking change, since packages had always been told not to call the Core.Compiler functions.

[nsajko]

Both Tuple and Union are part of the public API. So it seems like Tuple{Union{}} is public API, regardless of any Core.Compiler internals, no?

[aplavin]

was not a breaking change, since packages had always been told not to call the Core.Compiler functions.

Not sure why Core.Compiler.return_type is being brought up here over and over, it's completely unrelated to the issue.

[aplavin]

#51950 can be a "duplicate" from the internals PoV (function signatures are implemented as tuples), but for a user these are two different scenarios.
f(::Union{}) = ... mentioned in #51950 is a method that cannot be called at all (right?), while Tuple{Union{}} can sometimes arise in generic code without any bugs or internals usage.

[palday]

Making it an error--for this case which was already buggy--was not a breaking change, since packages had always been told not to call the Core.Compiler functions.

@vtjnash I've opened a PR against Dictionaries.jl that only uses public API and this problem still arises, so this is definitely not an issue arising relying on compiler internals.

[andyferris]

Union{} is always an interesting case, and given that Tuple parameters are covariant I can see that the behavior of Tuple{Union{}} might be different than Vector{Union{}} or wherever the parameter is invariant. E.g. it might be OK to instantiate an empty Vector{Union{}} but not a Tuple{Union{}} (which would be similar to trying to instantiate a Union{}, which is plainly impossible).

Still - in terms of the implementation, having Tuple{Union{}} become Union{} seems better for users than a runtime error. Especially for generic code.

As for Core.Compiler.return_type, I'm honestly happy to use whatever works. Note that sometimes whatever logic is used in Base in methods such as map(f, ::Vector) needs to be replicated in packages for other data structures. In Julia 1.10 this ultimately uses Base.@default_eltype to infer the eltype (which itself uses Core.Compiler.return_type).

[LilithHafner]

This seems like legitimate Julia code that does not depend on internals, though I don't think I've ever had a use-case for Union{}[] so I don't know how reasonable this usage is.

julia> eagerzip() = error()
eagerzip (generic function with 1 method)

julia> function eagerzip(args::AbstractArray...)
           allequal(axes.(args)) || throw(DimensionMismatch())
           Base.require_one_based_indexing(args...)
           res = similar(first(args), Tuple{eltype.(args)...})
           for i in eachindex(args...)
               res[i] = getindex.(args, i)
           end
           res
       end
eagerzip (generic function with 2 methods)

julia> eagerzip([1,2,3], [5,6,7])
3-element Vector{Tuple{Int64, Int64}}:
 (1, 5)
 (2, 6)
 (3, 7)

julia> eagerzip(Int[], [])
Tuple{Int64, Any}[]

julia> eagerzip(Union{}[], [])
Tuple{Union{}, Any}[] # 1.9
ERROR: Tuple field type cannot be Union{} # 1.10
Stacktrace:
 [1] eagerzip(::Vector{Union{}}, ::Vararg{AbstractArray})
   @ Main ./REPL[201]:4
 [2] top-level scope
   @ REPL[204]:1

[KristofferC]

Looks effectively the same as the example in #52385 (comment) (except much longer)?

[LilithHafner]

Yep! It's the same, just with a bit more motivation. The first example in the OP was, by itself, a valid regression report IMO.

[vtjnash]

Do you have an example of that? Arguably Union{}[] would have been the correct return there (which is what we fixed out to return in other code where it came up)

[LilithHafner]

No, I don't have an example of this breaking anything "in the wild" that does not depend on internals. If nobody else has such an example either, then we can call it a minor change and be done with it, but it is technically breaking.

[jariji]

Irrespective of semantic versioning, I don't see how it follows from any of the documented type rules that this should fail, so the error would be a special case, which isn't great from a user perspective.

[vtjnash]

it follows from any of the documented type rules that this should fail

It follows from the rule that the intersection of Tuple{T} and Tuple{S} is empty if the intersection of T and S is empty. That is a rule that subtyping has always had, so that is why this wasn't a major breaking change, as it only sets out to align the rest of the system with the existing rule. As for being a special case, this implementation is equivalent to adding a lower bound on the typevar for Tuple, which is not a particularly special case. Although it does also happen to implement the oft-requested feature of being able to prohibit a type var from being exactly Union{} although without yet making that feature very generally accessible.

but it is technically breaking

OT, but every change, including bugfixes, are technically breaking. But semantic versioning is mostly about not changing the result beyond the limits of what was promised. I actually wonder if there is an argument to be made that most changes from semi-working -> error or error -> working is allowable by semantic versioning therefore, since in neither case does a working program get a different return value.

[aplavin]

No, I don't have an example of this breaking anything "in the wild" that does not depend on internals. If nobody else has such an example either, then we can call it a minor change and be done with it

Aren't there enough examples in this thread already? I'm not sure why "internals" are even brought up here, as Core.Compiler is completely unrelated to this issue.

Julia docs say that

As per SemVer, code written for v1.0 will continue to work for all future LTS and Stable versions.

I've always interpreted that (and want to continue doing so...) that any code (not relying on internals/experimental) will continue to work. Not "only code that julia devs explicitly approve".

Moreover, as @jariji also points out, this is not just a breaking change in the abstract – it breaks totally sensible behavior, not a weird historical quirk.

Vector or AbstractVector with eltype == Union{} is a perfectly fine thing in Julia. It's also a natural thing to put such a vector into a StructArray. And this doesn't work anymore in 1.10.

Nothing in the Tuple/NamedTuple documentation suggests that they only support a subset of Julia types. Like, I can create another type with any parameter that I want:

julia> struct S{T}
       t::T
       end

julia> S{Union{}}
S{Union{}}

but not Tuple or NamedTuple for some reason.

[tpapp]

@aplavin: I think that the manual should just

1. explain the normalization of Tuple{...,Union{},...} to Union{} (following this change), with maybe a note about the pre-v1.10 behavior, if users encounter older code (but IMO this corner case is pretty rare in user code, most users rarely ever see Union{}),
2. explain that Union{} is a subtype of every other type, so eg foo(::Vector{<:T}) and foo(::Vector{T}) are different even when T is a concrete type (users run into this, see various discussions on the forum),
3. explain that Union{} is a valid type parameter, and code that accepts parametric types where this is relevant should be written accordingly.
4. provide an alternative suggestion to the "type vararg" that people used Tuple{...,Union{},...} for. In a lot of cases this is simply acknowledging the normalization and designing accordingly (eg "widening" accumulators), or if really necessary lifting (eg struct Wrap{T} end; Tuple{Wrap{Union{}},Wrap{Float64}}).

Of the above, 2. is not related to this issue or #49111, but I think it belongs in a section about Union{}.

[aplavin]

What plotting library is scatter from there — and what's the failure mode?

For example, let's take Makie – but this just an example, lots of functions in the ecosystem dispatch on the eltype.
I think, the failure would be same as this one we can simulate now:

using CairoMakie
scatter(Union{}[])

MethodError: plottype(::Vector{Union{}}) is ambiguous.
Candidates:
plottype(::AbstractVector{<:GeometryBasics.LineString})
@ Makie ~/.julia/packages/Makie/VRavR/src/interfaces.jl:217
plottype(::AbstractVector{<:GeometryBasics.AbstractPolygon})
@ Makie ~/.julia/packages/Makie/VRavR/src/interfaces.jl:222
Possible fix, define
plottype(::AbstractVector{Union{}})

Also, see all these functions: https://juliahub.com/ui/Search?q=AbstractVector{%3C:(Named)?Tuple}&type=code&r=true and more.

[mbauman]

Ah, that's great — and perhaps the best possible failure mode here. I see your this-is-a-regression path here and I respect the drum you're beating on, but I think it's worth noting how this is really on the edge of just-barely-working in multiple respects:

• If you were to construct the points slightly differently — perhaps by push!!-ing tuples incrementally — you'd hit the same ambiguity error in scatter.
• If you were to construct the points slightly differently — perhaps with a comprehension or broadcast with a type instability — you hit a different error path with an Any[] array. Looooong ago we very briefly had empty comprehensions return Union{}[], but that was too problematic at the time. The fact that using a StructArray in this very particular case happens to skate past some of that problematicness with a Tuple{Union{}} eltype is interesting... but the waters it's skating over are even more shark-infested than the Union{}[] alone.
• If that plotttype method table were just slightly different — perhaps more directly dispatching on <:Tuple like the methods you highlighted with that search — then you'd be getting even closer to the buggy MWE I posted above.
• Union{} <: Tuple, so — without ambiguities from other methods — all those highlighted methods should continue to dispatch without trouble if RFC: allow Tuple{Union{}}, returning Union{} #53452 gets through. Of course that's a very big caveat, but at least the error message itself documents the fix.

Of course, just-barely-working is still working, and none of the above points mean this isn't fundamentally a change in previously-working behaviors. How we best (potentially retroactively) communicate this change is a good question. But I'm inclined to wait for the dust to settle with #53516 and #53452 before figuring that step out.

[bvdmitri]

Why should the code

using CairoMakie
scatter(Union{}[])

even work in the first place? If any, I would expect it to throw an error sooner rather than later, and in fact, it does throw an error in 1.9 with the same ambiguity error. It's reasonable that this code doesn't function, as it is analogous to scatter(String[]), which also throws errors in both Julia versions.

As an example, the following code also throws errors in both 1.9 and 1.10:

x = String[]
# ... fill x, but it can remain empty ...
points = StructArray((eachindex(x), x))
scatter(points)

What makes Union{}[] special in a way that it must work in this scenario, especially when it's unclear how to handle it properly in plotting? A potential workaround would be same as handling x = String[] and could involve checking the eltype of x before attempting to plot or use it, particularly if one expects that the potential eltype of x could be somewhat unconventional/undesirable (although such situations are likely infrequent).

[aplavin]

Why should the code ... even work in the first place?

It shouldn't, and was specifically crafted to produce an error message now, that'll probably be similar to what

x = Union{}[]
# ... fill x, but it can remain empty ...
points = StructArray((eachindex(x), x))
scatter(points)

will produce when those linked PRs are merged.

the following code also throws errors in both 1.9 and 1.10
...
What makes Union{}[] special

The same code with Union{} worked on 1.9-. And this makes Union{} no more special than any other type, just the opposite: 1.10 made it "more special".

[aplavin]

Union{} <: Tuple, so — without ambiguities from other methods — all those highlighted methods should continue to dispatch without trouble

Indeed, the better link is https://juliahub.com/ui/Search?q=AbstractVector{%3C:(Named)?Tuple{&type=code&r=true. They won't probably work.

[aplavin]

Just checked - this regression appears to be present on 1.11 betas as well.

Over time, I've encountered more and more instances of completely innocent code broken in 1.10 and 1.11b. Union{} is such a natural eltype for empty arrays when nothing else is known about the array – many packages use it!

Here are a few more examples of code working in 1.9 but broken in 1.10.
1:

# if file content is {"a": [1], "b": [2]}, works on all Julia versions
# if file content is {"a": [], "b": []}, broken on 1.10 and 1.11
J = JSON3.read("somefile.json")
A = StructArray((J[:a], J[:b]))
scatter(A)

2:

# if "x=[1]\ny=[2]", works on all Julia versions
# if "x=[]\ny=[]", broken on 1.10 and 1.11
T = TOML.parsefile("somefile.toml")
A = StructArray((T["x"], T["y"]))
scatter(A)

I hope these examples would serve to further strengthen the motivation to find way to fix this regression – and not to switch those specific packages to return Vector{Any} instead of Vector{Union{}} :) Clearly using Any here is worse for many reasons. Also, there is surely more code using Vector{Union{}} aside from these packages, as this is the recommended start of the mutate-or-widen pattern.

I understand that this issue is nontrivial, but still keeping some hope that it can be fixed and these examples can run again in some future Julia version...

[vtjnash]

It seems you should be filing these bugs with StructArrays, since that seems to be where all the issues you keep finding are in

[aplavin]

Why, can you elaborate a bit more?

It doesn't seem to be using any internals, and not even the only popular package affected by this regression in Julia.
StructArrays is just the example I'm most familiar with. The exact same issue is manifested with eg TypedTables.Table.

With so many packages/functions using/returning arrays with Union{} eltype, it's really unfortunate to have this type a second-class citizen in Julia 1.10.