Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move all optimizers to Optimisers.jl #9

Merged
merged 15 commits into from
Feb 8, 2021
Merged

Move all optimizers to Optimisers.jl #9

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d7c53ac
Initial refactor
darsnack Jan 28, 2021
a7d2a36
Add remaining optimizers
darsnack Jan 28, 2021
5095550
Remove mutability, generic exports, fix style. Rename Sequence.
darsnack Jan 29, 2021
a5fa2a3
Fix typo in export
darsnack Jan 29, 2021
26d2c13
Revert interface.jl completely.
darsnack Jan 30, 2021
982f6bb
Comments from review
darsnack Jan 30, 2021
2f35662
Fix eps type
darsnack Jan 30, 2021
3f71db0
Revert constructors to positional
darsnack Jan 31, 2021
4320b7b
Switch to tuples for state instead of named tuples
darsnack Jan 31, 2021
1838870
Update to use f32 defaults
darsnack Feb 2, 2021
f2295d2
Update name to ChainOptimiser
darsnack Feb 4, 2021
26cf37d
Fix typo in ADAMW
darsnack Feb 4, 2021
dc83e2f
Rename to OptimiserChain
darsnack Feb 5, 2021
5da9dd8
Fix stack overflow for mismatched constructor types
darsnack Feb 5, 2021
ce489fb
Move all tests from Flux to Optimisers
darsnack Feb 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions src/Optimisers.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ using Functors: functor, fmap, isleaf
include("interface.jl")
include("rules.jl")

export Descent, Momentum, Nesterov, RMSProp,
ADAM
export Descent, ADAM, Momentum, Nesterov, RMSProp,
ADAGrad, AdaMax, ADADelta, AMSGrad, NADAM, ADAMW, RADAM, OADAM, AdaBelief,
WeightDecay, SequenceOptimiser

end # module
12 changes: 5 additions & 7 deletions src/interface.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
function patch(x, x̄)
return x .- x̄
end
patch(x, x̄) = x .- x̄

function state(o, x)
if isleaf(x)
return init(o, x)
else
x, _ = functor(x)
map(x -> state(o, x), x)
return map(x -> state(o, x), x)
end
end

Expand All @@ -20,11 +18,11 @@ function update(o, x::T, x̄, state) where T
if x̄ === nothing
return x, state
elseif isleaf(x)
_update(o, x, x̄, state)
return _update(o, x, x̄, state)
else
x̄, _ = functor(typeof(x), x̄)
x, re = functor(typeof(x), x)
x, restructure = functor(typeof(x), x)
xstate = map((x, x̄, state) -> update(o, x, x̄, state), x, x̄, state)
re(map(first, xstate)), map(x -> x[2], xstate)
return restructure(map(first, xstate)), map(x -> x[2], xstate)
end
end
Loading