Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reimplementation of extensions to OpenACC region in time integration split routine #4954

Merged

Conversation

brobey
Copy link
Contributor

@brobey brobey commented May 17, 2022

Reimplementation of extensions to OpenACC region to encompass the entire if blocks for the split explicit step. The prior pull request #4853 was corrupted when it was rebased to the changes in the head version. The reimplementation is done with a more recent head version to minimize difficulties with the merges of the changes

  • Extending the OpenACC region to encompass all of the large iteration if block
  • Extending OpenACC region outside the if splitExplicitStep blocks
  • Fixing loop in diagnostic solve to run on GPU
    These changes are separate and independent from the addition of the OpenACC top of the split routine changes.

QU240 test passes on Darwin with NVHPC (ver 21.11), GCC (ver 9), and Intel (ver 19) builds. Test suite on Cori Haswell with the Intel compiler passes on the tests that pass with the head version (3 fail with the head version). I still don't have access to Chrysalis or Summit to test on those platforms. Changes are bit-for-bit identical.

@philipwjones philipwjones added mpas-ocean BFB PR leaves answers BFB OpenACC labels May 18, 2022
@philipwjones philipwjones self-requested a review May 18, 2022 19:16
Copy link
Contributor

@philipwjones philipwjones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve based on code inspection and verified that the results are bfb with master on Summit in GPU mode and on Chrysalis for the full PR test suite.

!$acc enter data copyin(layerThicknessCur)
!$acc enter data copyin(layerThicknessTend)
!$acc enter data copyin(normalBaroclinicVelocityCur)
if (varinp_compute_active_tracer_budgets) then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@brobey I did a test merge into master, and when I compiled I get this error:

mpas_ocn_time_integration_split.f90:1705:52:

 1705 |             if (varinp_compute_active_tracer_budgets) then
      |                                                    1
Error: Symbol ‘varinp_compute_active_tracer_budgets’ at (1) has no IMPLICIT type
make[3]: *** [mpas_ocn_time_integration_split.o] Error 1

Were all these varinp variables removed in another PR? It might be easiest to `

git rebase origin/master

and then recompile and fix these.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the other merge went in before this, so it just needs to be rebased.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rebased to the E3SM project master and changed the varinp variables to config variables. The branch now compiles and passes on Darwin with OpenACC.

@brobey
Copy link
Contributor Author

brobey commented May 26, 2022 via email

@mark-petersen mark-petersen force-pushed the brobey/mpas-ocean/openacc-newextendsplit branch from 99c4fe9 to d2f2da0 Compare May 26, 2022 21:36
@mark-petersen
Copy link
Contributor

@brobey I rebased on e3sm-project/master, and that version passes stand-alone nightly suite with gnu debug and optimized, and matches bfb with master.

Since I rebased the code, please take a look through and run your standard test, to make sure it came through as expected.

@brobey
Copy link
Contributor Author

brobey commented May 31, 2022

I had rebased with e3sm-project/master, so the additional rebase only had a few small changes. I reviewed these changes by inspection and there should be no impact.

@brobey
Copy link
Contributor Author

brobey commented May 31, 2022

Tested on Darwin with OpenACC and passes bfb with rebase.

@philipwjones
Copy link
Contributor

philipwjones commented May 31, 2022

Also double-checked on summit GPUs w/ OpenACC after rebase and still bfb there. Looks good.

@mark-petersen mark-petersen self-requested a review June 7, 2022 14:04
Copy link
Contributor

@mark-petersen mark-petersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This is ready to go.

jonbob added a commit that referenced this pull request Jun 7, 2022
…4954)

Reimplementation of extensions to OpenACC region in time integration split routine

Reimplementation of extensions to OpenACC region to encompass the entire
if blocks for the split explicit step:
* Extending the OpenACC region to encompass all of the large iteration
  if block
* Extending OpenACC region outside the if splitExplicitStep blocks
* Fixing loop in diagnostic solve to run on GPU
These changes are separate and independent from the addition of the
OpenACC top of the split routine changes.

[BFB]
@jonbob
Copy link
Contributor

jonbob commented Jun 7, 2022

Test merge passes:

  • ERS.ne11_oQU240.WCYCL1850NS.chrysalis_intel
  • SMS_D_Ld3.T62_oQU120.CMPASO-IAF.chrysalis_intel
  • SMS_D_Ld1.ne30pg2_r05_EC30to60E2r2.WCYCL1850.chrysalis_intel
  • PEM_Ln9.ne30pg2_EC30to60E2r2.WCYCL1850.chrysalis_intel

merged to next

@jonbob jonbob merged commit 97f3445 into E3SM-Project:master Jun 8, 2022
@jonbob
Copy link
Contributor

jonbob commented Jun 8, 2022

merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BFB PR leaves answers BFB mpas-ocean OpenACC
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants