Checkpointing, NetCDF, and golden master tests

[ali-ramadhan]

Call them system tests, model verification tests, or end-to-end tests, that's what this PR is about.

Currently a work in progress as I hit a roadblock with checkpointing so the tests will fail. See #141. BUT, for testing we can just set model.forcings = nothing and manually reconstruct the forcings as we know them.

Tests implemented so far:

1. Checkpointing integration test: Run two coarse rising thermal bubble simulations and make sure that when restarting from a checkpoint, the restarted simulation matches the non-restarted simulation numerically.
2. NetCDF output integration test: Run a coarse thermal bubble simulation and save the output to NetCDF at the 10th time step. Then read back the output and test that it matches the model's state.
3. Thermal bubble golden master test: Run the coarse thermal bubble simulation for 10 time steps and check that the model output matches a golden master output.
4. Deep convection golden master test

@jm-c @christophernhill: do post any ideas for tests that should be implemented in this PR.

[codecov]

Codecov Report

Merging #140 into master will increase coverage by 3.21%.
The diff coverage is 77.77%.

@@           Coverage Diff            @@
##           master   #140      +/-   ##
========================================
+ Coverage   56.78%    60%   +3.21%     
========================================
  Files          19     19              
  Lines         597    620      +23     
========================================
+ Hits          339    372      +33     
+ Misses        258    248      -10

Impacted Files	Coverage Δ
src/models.jl	80% <ø> (ø)	⬆️
src/output_writers.jl	37.64% <77.77%> (+37.64%)	⬆️
src/time_steppers.jl	67.72% <0%> (+0.63%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 47b4613...b880a9b. Read the comment docs.

[codecov]

Codecov Report

Merging #140 into master will increase coverage by 9.94%.
The diff coverage is 78.94%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
+ Coverage   56.03%   65.97%   +9.94%     
==========================================
  Files          19       19              
  Lines         605      629      +24     
==========================================
+ Hits          339      415      +76     
+ Misses        266      214      -52

Impacted Files	Coverage Δ
src/models.jl	80% <ø> (ø)	⬆️
src/output_writers.jl	79.78% <78.94%> (+79.78%)	⬆️
src/time_steppers.jl	67.72% <0%> (+0.63%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7163560...371af73. Read the comment docs.

[glwagner]

Why do we save FFT plans?

It makes sense that a user should plan new FFTs if they are going to run a model, even if they are restarting a checkpoint on the same computer.

@christophernhill what do you think?

[ali-ramadhan]

We don't. We set model.poisson_solver = nothing before checkpointing then reconstruct the FFT plans afterwards.

So you're saying the user should be manually planning the FFTs themselves?

[ali-ramadhan]

Note to self: Seems like the golden master tests fail on Mac (Travis CI) so it seems that JLD might not be portable (although I generated the JLD checkpoint files on Windows and the Linux tests pass).

So for golden master testing I should rely on NetCDF output which is guaranteed to be portable.

EDIT: Hmmm, it might actually just be finicky (sometimes passes, sometimes doesn't). Another reason to use NetCDF. But so far if I keep running the tests on my laptop it always passes.

[ali-ramadhan]

@jm-c and I discussed this PR yesterday and seemed good to merge. Looks like the tests work now so I'm merging.