Fix serial-MPI non-reproducibility for gswp3 CASA-CNP configuration #567
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SeanBryan51
force-pushed
the
fix-serial-mpi-non-reproducibility-for-casa-cnp
branch
2 times, most recently
from
a0c4c5d to
877e0d9
Compare
SeanBryan51
changed the title
SeanBryan51
force-pushed
the
fix-serial-mpi-non-reproducibility-for-casa-cnp
branch
from
877e0d9 to
cf11792
Compare
This is done to avoid exceptions due to uninitialised memory access for debug builds.
Currently the phen type is not initialised properly in the MPI implementation and results in uninitialised values being written to the restart file. This change initialises the phen type on allocation so it is initialised for both serial and MPI applications.
Some CASA variables required in the output file are not being communicated back to the master process from the workers. This change communicates the required variables from worker to master and is required to restore bitwise reproducibility in the CASA netcdf output file between serial and MPI runs.
Currently the MPI implementation does not output time-averaged pools and fluxes (#566). This change implements the time-averaging functionality which exists in the serial driver into the MPI master driver. This is required to restore bitwise reproducibility in the CASA netcdf output file across serial and MPI runs.
SeanBryan51
force-pushed
the
fix-serial-mpi-non-reproducibility-for-casa-cnp
branch
from
cf11792 to
6e609f4
Compare
|
Initialisation of CASA types was added in #590. Need to rebase. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently running serial and MPI runs for the gswp3 configuration (see MPI and serial configurations) with CASA-CNP enabled1 shows bitwise differences between serial and MPI runs in the CASA restart and CASA NetCDF output file (all other outputs, e.g. standard CABLE outputs and restarts, are bitwise identical between serial and MPI). This change fixes a few bugs in the MPI master driver and the CASA-CNP code so that we have bitwise reproducibility between serial and MPI in the CASA output and restart files for this configuration.
Type of change
Please delete options that are not relevant.
Checklist
Testing
Please add a reviewer when ready for review.
📚 Documentation preview 📚: https://cable--567.org.readthedocs.build/en/567/
Footnotes
Note: CASA-CNP was enabled without a CASA restart file (i.e.
cable_user%CASA_fromZero = .TRUE.). ↩