Improve documentation on custom broadcast.

[MasonProtter]

Currently, I find the documentation on customizing broadcast fairly hard to read, especially if you are implementing something that has very different behaviour from standard array broadcast.

I think one of the core missing things from the documentation is that it doesn't help a user understand what actually happens to their code when they write f.(xs...), and it seems julia doesn't provide any tools for figuring that out (something analogous to macroexpand or even @which for broadcast calls would be nice if it doesn't already exist tucked away somewhere).

After spending some time digging in Base.broadcast.jl and looking at some other people's implementation code I came to understand that when I write f.(xs...), first Base.broadcastable will get mapped across xs and then Broadcast.combine_styles will be applied to that result. Broadcast.combine_styles will then apply Base.BroadcastStyle to your arguments and then do a bunch of logic to try and figure out a common BroadcastStyle for all the types. Then there will be a call like

broadcasted(combine_styles(broadcastable(xs)), f, xs...)

and then materialize will get called on that.

I think at least a rough understanding of this pipeline is necessary in order to utilize the docs on customizing this process, so I think it should be spelled out a bit more clearly somewhere. I'd offer to try and write something, but honestly I don't feel qualified. My understanding is still quite shaky.

Apologies if this is actually spelled out somewhere and I just missed it.

[mbauman]

Improvements here are always great. A part of the tension is that I want to document behaviors and not the exact implementation.

You may be looking for Meta.@lower — and perhaps we should show this in the documentation:

julia> Meta.@lower f.(x, y, z)
:($(Expr(:thunk, CodeInfo(
    @ none within `top-level scope'
1 ─ %1 = Base.broadcasted(f, x, y, z)
│   %2 = Base.materialize(%1)
└──      return %2
))))

[MasonProtter]

Ah, you are right, Meta.@lower is what I was looking for. Perhaps even a suggestion in the docs to use Meta.@lower and @which to trace out what's happening would be helpful? Also, maybe the implementation could be documented in the Developer Docs?

Personally, I found the suggestions in customizing broadcast unhelpful until I had at least a rough understanding of why I needed to do that and how the methods I was told to define would be used. Without that understanding, you don't really know which overloads and definitions are even relevant to your use-case.

[MasonProtter]

For instance, I thought I was following the suggestions well but I couldn't figure out why my code wasn't working until I realized that I actually needed to define broadcastable for my type and without that it was just treating my code as scalar. Part of that confusion was the listing of broadcastable as optional and the description that it should "Convert x to an object that has axes and supports indexing" which didn't seem relevant to me since the broadcast implementation for my type didn't involve indexing.

[mbauman]

The other part of the problem is that there are so many ways one can want to customize broadcasting. What were you trying to do?

[MasonProtter]

Well, this is a bit embarrassing but I was just trying to understand broadcast better, so I tried to (ab)use it to implement a suggestion I saw on Slack where f.(g) means f ∘ g. I ended up coming up with this:

struct CompositionStyle <: Broadcast.BroadcastStyle end
Broadcast.broadcasted(::CompositionStyle, f, g) = f ∘ g

macro compose_castable(fs...)
    out = Expr(:block)
    for f in fs
        ex1 = :(Base.BroadcastStyle(::Type{typeof($f)}) = CompositionStyle())
        ex2 = :(Base.broadcastable(::typeof($f)) = $f)
        push!(out.args, ex1, ex2)
    end
    esc(out)
end

f(x) = 2x + 1
g(x) = (x^2)/2
h(x) = log(2x)/5

@compose_castable g h

julia> f.(g).(h)(2)
1.0768724822269122

[mbauman]

I hope you can appreciate that this (ab)use case isn't the target audience of that documentation section. :)

There's definitely room for a devdocs section that details the implementation and is targeted at other Julia core developers, but I really don't want to encourage using dots to mean anything other than broadcast.

[MasonProtter]

Yes of course, as I said, it was for educational purposes so I could learn how to implement custom broadcast. I imagine there are good examples out there though that we could draw on which implement a meaningful form of broadcast on a type that's not very much like an Array though.

[apkille]

I would be open to submitting a PR to improve the documentation myself. It's no secret that defining a custom broadcast interface is involved and extremely hard for the majority of Julia users. I spent quite a long time deciphering the current documentation (https://docs.julialang.org/en/v1/manual/interfaces/#man-interfaces-broadcasting) and scanning source code in JuliaArray packages to understand how it's actually implemented.

The other part of the problem is that there are so many ways one can want to customize broadcasting. What were you trying to do?

Maybe more than one example should be included in the documentation then. IMO, it should also be clear how to define an interface for a concrete type that is not a subtype of AbstractArray but wraps around AbstractArrays (@MasonProtter alluded to this).