Enable GPU execution of MPAS-Atmosphere scalar transport through OpenACC #1176
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tives This commit adds code and OpenACC directives in the MPAS_atm_dynamics_init routine to transfer invariant fields used by scalar transport code to the device, and it adds code and directives in the MPAS_atm_dynamics_finalize routine to delete these invariant fields. The new code and directives are surrounded by tests on the MPAS_OPENACC preprocessing macro, and so the changes in this commit have no effect unless MPAS-A is compiled with OPENACC=true. Since the MPAS_atm_dynamics_init routine is called once during model startup and the MPAS_atm_dynamics_finalize once during model shutdown, the invariant fields will remain resident on the device across all dynamics time steps.
This commit adds OpenACC directives throughout the atm_advance_scalars_work routine. Also as part of this commit, macros have been defined at the top of mpas_atm_time_integration.F to activate timers for OpenACC data transfers when MPAS-A is compiled with OPENACC=true. For the data transfers in the atm_advance_scalars_work routine, the new timer atm_advance_scalars [ACC_data_xfer] will be written to the timer summary in log files. With the NVHPC 24.3 compilers and flags provided in the 'nvhpc' target, results for a regional test case are bitwise identical when the original code and code in this commit are both compiled with OPENACC=true.
This commit adds OpenACC directives throughout the atm_advance_scalars_mono_work routine. At present, halo exchanges in the monotonic scalar transport routine require fields to be transferred to the host before the halo exchange, and then back to the device afterward. For the data transfers in the atm_advance_scalars_mono_work routine, the new timer atm_advance_scalars_mono [ACC_data_xfer] will be written to the timer summary in log files. With the NVHPC 24.3 compilers and flags provided in the 'nvhpc' target, results for a regional test case are bitwise identical when the original code and code in this commit are both compiled with OPENACC=true.
gdicker1
approved these changes
May 30, 2024
20 tasks
gdicker1
added a commit
to gdicker1/MPAS-Model
that referenced
this pull request
Jul 30, 2024
This commit ensures the invariant fields used during this work routine are present on the device from model startup to model shutdown. It builds on the changes in PR MPAS-Dev#1176 to copyin invariant fields during mpas_atm_dynamics_init and delete them from the device during mpas_atm_dynamics_finalize.
mgduda
added a commit
that referenced
this pull request
Jul 31, 2024
This merge enables GPU execution of the atm_compute_solve_diagnostics_work routine through the addition of OpenACC directives. Timing information for the OpenACC data transfers in this routine is captured in the log file by a new timer: atm_compute_solve_diagnostics [ACC_data_xfer]. As in #1176, invariant fields are copied in during mpas_atm_dynamics_init and deleted in mpas_atm_dynamics_finalize. However, these do not minimize the data movement as atm_compute_solve_diagnostics is called during the initialization phase before mpas_atm_dynamics_init is called. * atmosphere/acc_solve_diagnostics: Add acc data management directives to atm_compute_diagnostics_solve_work Add acc parallel and loop directives to port atm_diagnostics_solve_work Prepare atm_compute_solve_diagnostics_work for porting Add acc data directives for invariant arrays used in atm_compute_solve_diagnostics
gdicker1
added a commit
to gdicker1/MPAS-Model
that referenced
this pull request
Aug 29, 2024
This commit ensures the invariant fields used during this work routine are present on the device from model startup to model shutdown. It builds on the changes in PR MPAS-Dev#1176 to copyin invariant fields during mpas_atm_dynamics_init and delete them from the device during mpas_atm_dynamics_finalize.
gdicker1
added a commit
to gdicker1/MPAS-Model
that referenced
this pull request
Dec 6, 2024
Ensures the fields which don't change while the model is running are present on the device from model startup until model shutdown. It builds on the changes in PR MPAS-Dev#1176 to copyin invariant fields during mpas_atm_dynamics_init and delete them from the device during mpas_atm_dynamics_finalize.
gdicker1
added a commit
to gdicker1/MPAS-Model
that referenced
this pull request
Jan 7, 2025
Ensures the fields which don't change while the model is running are present on the device from model startup until model shutdown. It builds on the changes in PR MPAS-Dev#1176 to copyin invariant fields during mpas_atm_dynamics_init and delete them from the device during mpas_atm_dynamics_finalize.
gdicker1
added a commit
to gdicker1/MPAS-Model
that referenced
this pull request
Jan 10, 2025
Ensures the fields which don't change while the model is running are present on the device from model startup until model shutdown. It builds on the changes in PR MPAS-Dev#1176 to copyin invariant fields during mpas_atm_dynamics_init and delete them from the device during mpas_atm_dynamics_finalize.
gdicker1
added a commit
to gdicker1/MPAS-Model
that referenced
this pull request
Jan 16, 2025
This commit ensures the invariant fields used during this work routine are present on the device from model startup to model shutdown. It builds on the changes in PR MPAS-Dev#1176 to copyin invariant fields during mpas_atm_dynamics_init and delete them from the device during mpas_atm_dynamics_finalize.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR enables the scalar transport routines in MPAS-Atmosphere to run on GPUs through the addition of OpenACC directives. Specifically, the
atm_advance_scalars_workandatm_advance_scalars_mono_workroutines have been ported via OpenACC.Additionally, timing information for OpenACC data transfers in the scalar transport routines is captured and reported in the log file by two new timers:
atm_advance_scalars [ACC_data_xfer]atm_advance_scalars_mono [ACC_data_xfer]To minimize data movement, the MPAS-Atmosphere dycore has also been modified to copy all invariant fields needed by the scalar transport routines onto the device before the first dynamics timestep and to delete those invariant fields after the last dynamics timestep.