Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Checkpointer revival #326

Merged
merged 28 commits into from
Aug 28, 2019
Merged

Checkpointer revival #326

Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
1212c82
Checkpointer that saves the bare essentials using JLD2.
ali-ramadhan Aug 2, 2019
d434e55
Revive checkpointer test
ali-ramadhan Aug 2, 2019
40fd4b0
Merge branch 'master' into checkpointer
ali-ramadhan Aug 22, 2019
b47fda8
Do not save forcing functions.
ali-ramadhan Aug 22, 2019
f7476b4
Move output utils to the bottom.
ali-ramadhan Aug 22, 2019
171f582
Validate frequency and interval.
ali-ramadhan Aug 22, 2019
14c221b
Remove redundant warnings.
ali-ramadhan Aug 22, 2019
823d310
User can select kwargs to pass to restored model (e.g. for forcing).
ali-ramadhan Aug 22, 2019
7793b1e
Clean up checkpointer utils.
ali-ramadhan Aug 22, 2019
02cb710
Update checkpointer test.
ali-ramadhan Aug 22, 2019
ddbc1d6
Forgot to test checkpointer on GPU.
ali-ramadhan Aug 22, 2019
8aeacbd
Switch to saveproperty! for JLD2OutputWriter.
ali-ramadhan Aug 25, 2019
0f933c8
Analogous serializeproperty! methods for checkpointing.
ali-ramadhan Aug 25, 2019
ebb0f34
Saving/serializing boundary conditions.
ali-ramadhan Aug 25, 2019
6935cf6
Refactor Checkpointer.
ali-ramadhan Aug 25, 2019
c96175b
Properly restore fields and boundary conditions.
ali-ramadhan Aug 25, 2019
cbd4a8d
Index named tuples by name, rather than number/range.
ali-ramadhan Aug 26, 2019
64db4c7
hasfunction utility.
ali-ramadhan Aug 26, 2019
c2c01ee
Revert "Index named tuples by name, rather than number/range."
ali-ramadhan Aug 26, 2019
3552f77
Checkpointer will serialize bcs as a whole.
ali-ramadhan Aug 26, 2019
89518cf
Cleanup, comments, and update test.
ali-ramadhan Aug 26, 2019
cffc11d
Fix typo.
ali-ramadhan Aug 26, 2019
bcfaff8
Nuke HorizontalAverages and VerticalPlanes scratch structs.
ali-ramadhan Aug 26, 2019
ea91e5f
Simpler validate interval and frequency function
ali-ramadhan Aug 27, 2019
517b32b
Clean up checkpointer constructor
ali-ramadhan Aug 27, 2019
898216b
adds array_refs property to checkpointer and generalizes hasfunction
glwagner Aug 27, 2019
00b6eba
gets has_reference function working with a little elbow grease
glwagner Aug 27, 2019
4e72d35
moves using Distributed to Oceananigans.jl from output_writers
glwagner Aug 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions src/output_writers.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,19 +392,29 @@ VerticalPlanes(model) = VerticalPlanes(model.arch, model.grid)
mutable struct Checkpointer <: OutputWriter
dir :: String
prefix :: String
output_frequency :: Int
frequency :: Int
interval :: FT
structs :: S
fieldsets :: F
end

function Checkpointer(; output_frequency, dir=".", prefix="checkpoint", force=false)
function Checkpointer(; output_frequency, dir=".", prefix="checkpoint",
checkpointed_structs = (:arch, :boundary_conditions, :grid, :clock, :eos, :constants, :closure),
checkpointed_fieldsets = (:velocities, :tracers, :G, :Gp),
force=false)
mkpath(dir)
return Checkpointer(dir, prefix, output_frequency)
end

function savesubfields!(file, model, name, flds=propertynames(getproperty(model, name)))
for f in flds
file["$name/$f"] = Array(getproperty(getproperty(model, name), f).data.parent)
if name ∉ (:forcing)
Copy link
Member

@glwagner glwagner Aug 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused about how this function works:

  • why restrict this functionality to Field (as it looks to me), but provide a hook when name=:forcing?
  • can't we check the type of the object being saved? Other fields (like boundary conditions, or future fields that we add to model) may be functions.

I think it'd be better to use a function that dispatches on the type of the object being saved, eg

savefield(file, location, field) = file[location] = field
savefield(file, location, field::AbstractArray) = file[location] = Array(field)
savefield(file, location, field::Field) = file[location] = Array(field.data.parent)
savefield(file, location, field::Function) = warn("Cannot checkpoint Function object into $location!")

Using such a function may be more robust and general.

for f in flds
file["$name/$f"] = Array(getproperty(getproperty(model, name), f).data.parent)
end
else
@warn "Cannot save subfields of $name"
end
return nothing
return
end

checkpointed_structs = [:arch, :boundary_conditions, :grid, :clock, :eos, :constants, :closure]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These fields could be made properties of the Checkpointer. This will allow the user to add/remove fields from checkpointing, which may be generally useful in the future and is currently useful for omitting boundary conditions from checkpointing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ali-ramadhan was this resolved?

Expand Down