Inferrably construct leaf types

[timholy]

I just transitioned a large internal project from FixedSizeArrays->StaticArrays. I may have missed it, but the one bit of functionality I found lacking was the ability to easily & inferrably create a leaftype. I use this facility to allocate Array{SA} where SA <: StaticArray gets constructed from other type parameters. The leaf2 test illustrates how you might inferrably create an empty vector of NxN SMatrix with eltype Float32, given any function argument that gives you access to N as a type parameter:

leaf2{N}(::NTuple{N,Bool}) = Vector{SArray(Size(N,N), Float32)}(0)

It's just one line of code, but documenting and testing it seems to be the main thing here.

Again, maybe there's already a different way to do this, so if I missed it please do let me know.

[coveralls]

Coverage increased (+0.1%) to 79.972% when pulling 8eaccf5 on teh/leaftypes into 16f6c08 on master.

[c42f]

I don't think there's a way to do this currently, and what you've got here looks like a great solution to me.

Is it fair to say that this is mainly useful because writing the concrete type for SArray is a bit of a pain, particularly having the extra L parameter, and needing to wrap things in Tuple? (ie, we'd ideally like the SArray type to be written SArray{N,M,P,..., T} rather than SArray{Tuple{N,M,P,...},T,L}).

[andyferris]

The one interesting fact about this PR's implementation is that we call a type and don't get an instance of that type (it's not a constructor). Does this happen elsewhere in the language? E.g. we use real(Complex{Float64}) -> Float64 rather than Real(Complex{Float64}), I think for this reason.

OTOH it we already have:

julia> similar_type(SArray, Float64, Size(2,2))
StaticArrays.SMatrix{2,2,Float64,4}

and users can easily overload the similar_type behavior to suit. (EDIT: I do note that this won't work for e.g. Rotations)

[andyferris]

By the way, I think Base should implement similar_type for exactly the reason in the OP. How do you deal with this for dynamic arrays, Tim?

[c42f]

we call a type and don't get an instance of that type

Ok, good point :-/

similar_type does roughly the right thing here, but there's still the oddity with MMatrix vs SMatrix, where we are choosing to return SMatrix (for good reasons), but the user might be surprised by this if they're really just trying to have a more convenient way of writing the concrete type name.

[timholy]

Is it fair to say that this is mainly useful because writing the concrete type for SArray is a bit of a pain, particularly having the extra L parameter

Yes, that's precisely the issue. foo{M,N}(::NTuple{M}, ::NTuple{N}) = NTuple{M*N, Any} is not inferrable.

The one interesting fact about this PR's implementation is that we call a type and don't get an instance of that type (it's not a constructor). Does this happen elsewhere in the language?

There would be a certain sense in being able to write this as SArray{S::Size, T} and have it fill in N and L---any type where N != length(S) or L != prod(S) is nonsensical, so there's no advantage in having the parameters be apparently independent. But we can't do that now. JuliaLang/julia#22056

We already have similar_type

I knew there had to be a method for "spelling" this, I should have reread the README...

But I do think that when one wants a specific type, it would be better to be able to spell that out explicitly. Maybe one should implement this as sarray(Float64, S::Size)?

How do you deal with this for dynamic arrays, Tim?

It just doesn't tend to come up, perhaps because there aren't that many arrays with dependent type parameters. Or I build a single instance and then use Vector{typeof(x)}(n). But in my codebase I have so many places where I allocate an array-of-staticarrays, that I noticed the extra bit of ugliness in (apparently) not having it be easy to do this.

[KristofferC]

This is inferrable though:

julia> Base.@pure pure_mul(a::Int, b::Int) = a * b
pure_mul (generic function with 1 method)

julia> foo{M,N}(::NTuple{M}, ::NTuple{N}) = NTuple{pure_mul(M,N), Any}
foo (generic function with 1 method)

julia> @code_warntype foo((1,2), (3,4,5))
Variables:
  #self#::#foo
  #unused#@_2::Tuple{Int64,Int64}
  #unused#@_3::Tuple{Int64,Int64,Int64}

Body:
  begin 
      return NTuple{6,Any}
  end::Type{NTuple{6,Any}}

[timholy]

Indeed it is. I was just noting that it's an extra step that would be nice to not have to think about.

[KristofferC]

I wonder if some functions in Base could be special cased to be considered pure.. Like simple arithmetic on some leaf types (like Int). I guess that means you couldn't interactively redefine them but redefining 5+3 won't get you very far before a crash anyway.

[timholy]

There's also the "do more constant propagation during inference" approach (e.g., JuliaLang/julia#5560), but my understanding is that this is not a finite problem.

[timholy]

I can close this and use similar_type in my code, but I do think it would be better to have an explicit type-constructor. How should we proceed?

[c42f]

Does similar_type tick all the boxes? Seems to me that similar_type as it exists has a slightly different meaning:

julia> similar_type(MVector, Int, Size(2,))
SVector{2,Int64} # Now it's immutable

MVector is probably a bit of a sideshow for your immediate use case, but it seems like whatever we come up with should extend naturally to MVector as well.

I quite like what's here, with the exception that the SArray constructor is being used as a type constructor rather than a value constructor. Perhaps that's an acceptable pun, on the whole?

[andyferris]

Perhaps that's an acceptable pun, on the whole?

In my book, if it's OK in Base then it's OK here. I haven't seem this pun in Base, and I suspect some thought over the years has gone into the upper/lower case pairs like real/Real, and so-on.

Like Chris said, here sarray is unfortunate because you have to then have marray and sizedarray and so-on for user-defined types. It seems more Julian to have one function which is overloaded.

What we kind-of want is to overload apply_type (what the curly braces eventually get lowered to). In the absence of that, I'm guessing something like a leaf_type function would be OK. Again, my acceptance criteria would be that it would also be useful (or at least usable) for other AbstractArrays, with behavior such as leaf_type(a::Array, T, size::NTuple{N, Integer}) = Array{T, N}.

(Is "leaf" or "concrete" the usual term from the user side? I've always seen "visualize the type-tree" terms like leaf, bottom, top, etc as being more used internally by Julia devs, while the docs mainly distinguish abstract vs. concrete)

[c42f]

Ok @andyferris if you think it's a bad pun, I think we should go with similar_type here. It works for Tim's immediate use case if he only cares about SArray for now.

The large problem is that we've complicated the semantics of similar_type by making it turn MArray into SArray, as an optimization in this package. Is there a nice way we can fold a notion of changing the mutability into similar_type? A mutability argument (presumably set to Val{true} or Val{false}) would work but might be ugly.

[c42f]

By the way, in the future I could imagine having a somewhat more flexible apply_type, which could allow us to do away with the L and N parameters. If we also had curlies meaning Tuple construction, writing the full SArray type would no longer be onerous - we'd have

SArray{{2,2,2},Int}

instead of

SArray{Tuple{2,2,2},Int,3,8}

(of course, that's speculation at the moment.)

[c42f]

Tim, would similar_type work well enough for you? It's not as syntactically compact, but it's less of a somewhat weird pun on the constructor.

If similar_type is good enough, I suggest we close this.

[timholy]

It does for my immediate needs. We clearly need a better way to do this; https://github.com/vtjnash/ComputedFieldTypes.jl seems like a pretty good answer, though. It would be a big job to switch, so I'd be curious to know if it's of interest to you folks. (I might put together a PR, but not in the near future.)

[c42f]

Interesting, thanks for the link.

[timholy]

Hmm, don't rush to adopt ComputedFieldTypes just yet: vtjnash/ComputedFieldTypes.jl#2.