Checkpoint/Resume

[thorstenhater]

What is this?

Implement (partial!) dumping of simulation state through a serialization object that can be use to reset the simulation
to an earlier state. The produced snapshot includes just enough information to perform that reset,
not all bits.

For a successful resume, the following constraints apply:

• context and domain_decomposition must be identical to those used during serialize
• the simulation we call resume on must be
  • valid
  • constructed using the same recipe

Not included

• pointer data that forms internal references.
• internal states of generators, samplers, and probes. Might be needed to change in between snapshots.

⚠️ Breaking changes

I needed to bump CUDA language level to 17 (up from 14). That'll require CUDA 11+; released in 2020.

TODO

• GPU.
• At least partially check the constraints It's largely impossible. See docs. Suggestions welcome, though.
• Python bindings.
• Docs

Issues

Closes #560

[brenthuisman]

@boeschf doublechecks CUDA 11 avail on piz daint: it is!

[boeschf]

Wouldn't it be less error-prone if the base class template event_stream_base had its own serialization hooks which could be called from here?

[thorstenhater]

Hmm. It's a tradeoff. In the current way I can use the ENABLE macro in the multicore subclass
and need a custom serializer in the gpu case. Your proposal is slightly more code to write:

1. ENABLE in base
2. custom one-liner in multicore
3. custom serializer in gpu minus the base members plus one line for serializing base

The problem arises when writing out the event spans. One is a GPU memory view aka (ptr, length)
and the multicore is a range aka (ptr_beg, ptr_end). It seems prudent to maybe merge representations
first?

[boeschf]

yeah, I see your point. Let's leave it the way it is.

[boeschf]

cscs-ci run daint-gpu

[brenthuisman]

cscs-ci run daint-gpu

[brenthuisman]

cscs-ci run daint-gpu

[brenthuisman]

cscs-ci run daint-gpu

[brenthuisman]

Still an error in the distributed GPU job:

gap_junctions: srun: error: nid05416: task 1: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=48211655.1
slurmstepd: error: *** STEP 48211655.1 ON nid05415 CANCELLED AT 2023-08-07T17:55:22 ***
   - gap_junctions: srun: error: nid05415: task 0: Exited with exit code 143

[thorstenhater]

JUWELS disagrees

$ srun --account=XXXX --partition=XXXX bin/gap_junctions
gpu:      yes
threads:  1
mpi:      yes
ranks:    1

Using default parameters.
running simulation

30 spikes generated at rate of 3.33333 ms between spikes

---- meters -------------------------------------------------------------------------------
meter                         time(s)      memory(MB)
-------------------------------------------------------------------------------------------
model-init                      0.016           1.654
model-run                       1.098           1.409
meter-total                     1.114           3.062

[boeschf]

yeah, I see your point. Let's leave it the way it is.