Add Parallel layer

[darsnack]

Since #1289 stalled, I have added an implementation of Parallel with some of the changes we discussed during ML calls. This version excludes most of the structural layers in #1289 like Join, Split, and Nop. I also added the ability for the user to specify the reduction operator. If it is acceptable, I would like to remap SkipConnection to Parallel (not a deprecation exactly).

The reason for submitting this PR now is because I am creating pre-trained weights for the networks in FluxML/Metalhead.jl#70, and there is a lot of code that can be replaced with a Parallel. So, I'd like to have Parallel in Flux before continuing with training to make the process easier.

PR Checklist

• Tests are added
• Entry in NEWS.md
• Documentation, if applicable
• Final review from @DhairyaLGandhi (for API changes).

cc @CarloLucibello

[darsnack]

Also, I'd like to add that outputsize worked out of the box with no changes when I added Parallel. So that's a small win to celebrate.

[DhairyaLGandhi]

Thanks, do we also want to add the functionality from having multiple arguments in the forward pass as in parallel execution and reduction? Seems like there's some additional functionality in #1289

[darsnack]

Adding multiple input arguments would be like Join from #1289. We could add a Vararg version of Parallel that implements that behavior.

[darsnack]

Actually on second thought, maybe we shouldn't have the multi-arg version? It complicates the semantics on Chain, and I don't think it is used in practice for anything.

[darsnack]

@DhairyaLGandhi any decision on keeping vs excluding multiple input arguments for the layer? I would recommend excluding.

[DhairyaLGandhi]

The vararg AbstractArray method is the one I had in mind. I would also want to see some addition to the docs that talks about implementing the Join and Split because I feel we would inevitably have questions about wanting them.

Otherwise, lgtm

[darsnack]

What about SkipConnection being syntactic sugar for Parallel with identity on one of the paths?

[DhairyaLGandhi]

I'm in favour of retaining the struct because it's a very common case, but rewriting it in a follow up

[DhairyaLGandhi]

@darsnack I was hoping the docs would show how to implement Join in terms of Parallel.

[darsnack]

Note that Join is the same as Parallel with variable arguments. There isn't much work required to implement Join in terms of Parallel. So I just added a subsection to the docs. This should be more instructive for users.

[darsnack]

@DhairyaLGandhi bump

[DhairyaLGandhi]

Lgtm, thanks @darsnack

[DhairyaLGandhi]

bors r+

[darsnack]

@bhvieira Currently, it is still in Flux as it was before. My plan was not to deprecate it but to define:

SkipConnection(layers, connection) = Parallel(connection, layers, identity)

I think the SkipConnection syntax/nomenclature is still useful and meaningful.

[bhvieira]

@darsnack Yeah, I deleted my comment before seeing yours actually, that's what I had in mind! Good job btw!

[bors]

Build succeeded:

• buildkite/flux-dot-jl

[DhairyaLGandhi]

Can we add follow on testing on the GPU, that seems to have been missed? My bad. @darsnack