-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bring back store!
, accumulate
, and accumulate!
?
#113
Comments
actually #110 ends up removing |
Yeah. |
store!
?store!
, accumulate
, and accumulate!
?
I supect we want:
Also we should wait for mutation to be happening as that affects this signficantly. I am happy enough for now to let the AD solve this. And then if we find common patterns look at abstracting them |
* Add Logo * Add Logo to readme * fix max size of logo in readme * fix max size of logo in readme * fix max size of logo in readme * fix width in readme
Have thought more about it for accumulate!(x, y) = x + y
accumulate!(x::AbstractZero, y::InplaceThunk) = unthunk(y)
accumulate!(x, y::InplaceThunk) = (y.add!(x); x)
accumulate!(x::InplaceThunk, y) = accumulate!(y, x)
accumulate!(x::AbstractThunk, y::InplaceThunk) = accumulate!(unthunk(x), y) the extension to make sure it does the right thing, and is fast for multiple arguments is: @generated function accumulate!(args...)
# need to make sure optimized multiarg + gets hit for arrays etc.
# so need to group operations so we get `add!(add!(add!(n1+n2+n3, i1), i2), i3)`
out_of_place_elements = Expr[]
out_of_place_op = Expr(:call)
inplace_op = out_of_place_op # initially we will just have the out of place sum
for (ind, argT) in enumerate(args)
element = :(@inbounds arg[ind])
if argT == InplaceThunk
inplace_op = :(accumulate!($inplace_op, $element))
else
push!(out_of_place_elements, element)
end
end
if isempty(out_of_place_args)
#it is empty, so nothing to accumulate! against
out_of_place_op.args = [:Zero]
else # not empty so we have things to sum
out_of_place_op.args = [:+; out_of_place_elements]
end
return inplace_op
end basically we need to first do all the out of place accumulation in a single call to julia> const z = rand(1024, 1024);
julia> @btime z + z + z + z;
1.305 ms (2 allocations: 8.00 MiB)
julia> @btime +(+(+(z, z), z), z);
2.928 ms (6 allocations: 24.00 MiB) The fact that this is already fast is why we don't need to do special things with broadcast ourself for handling making sure multiple arrays are fast. Above code is not tested. We want this for Zygote to start using when it starts using JuliaDiff/ChainRules.jl#240 |
maybe
That seems like it might start to get complicated particularly with the multi-arg versions |
We removed
store!
in #110, but one day we might want something like that back.Paraphrasing @willtebbutt's comment
Just as it's useful to have
accumulate!(::Array, ::Array)
(i.e.add!
) to avoid accumulating differentials into a third array, it maybe useful to havestore!
for storing the result into a pre-allocated array. E.g. If you're a reverse-mode AD like ReverseDiff, andstore!
would allow you the option to pre-allocate all of the memory required for the forwards- and reverse- passes over a programme.for ease of reference, the old code was
The text was updated successfully, but these errors were encountered: