Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gpu improvements #66

Closed
wants to merge 30 commits into from
Closed

Gpu improvements #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
0df3f2a
improve handling of gpu options
jedwards4b Nov 1, 2022
a13d803
only provide gpu flags to specific files
jedwards4b Nov 3, 2022
09a6835
generalize these flags, they should not be casper specific, add openm…
jedwards4b Nov 3, 2022
ff2b2fe
correct typo, update cuda module
jedwards4b Nov 3, 2022
1d7974a
more cleanup of gpu functionality
jedwards4b Nov 3, 2022
fbc05d6
remove -gpu stuff
jedwards4b Nov 7, 2022
0a4702a
Merge remote-tracking branch 'origin/gust_update022423' into gpu_impr…
jedwards4b Mar 15, 2023
9beb292
Add Jim's changes for Gust's modules from https://github.com/jedwards…
sjsprecious Mar 22, 2023
4134af3
Add Jim's gpu_improvement branch at https://github.com/jedwards4b/ccs…
sjsprecious Mar 22, 2023
3b7a182
Add max_cputasks_per_gpu_node, gpu_type and gpu_offload options for C…
sjsprecious Mar 23, 2023
ee726b5
update batch script template for Gust GPU node
sjsprecious Mar 23, 2023
1f5eb30
typo fix for mismatched tags for GPU options
sjsprecious Mar 23, 2023
8649a95
Add missing "MAX_GPUS_PER_NODE"
sjsprecious Mar 24, 2023
e7bfbe1
downgrade netcdf-mpi fpr nvhpc/22.11
sjsprecious Mar 25, 2023
bf14967
bug fix of batch script template with gpu_enabled option
sjsprecious Mar 25, 2023
14f605e
remove -target=zen3 flag on gust
sjsprecious Mar 25, 2023
59086af
update to nvhpc/23.1 and related modules
sjsprecious Mar 26, 2023
918e3a8
turn off MPS on Gust
sjsprecious Mar 26, 2023
85045b9
updates from jian
jedwards4b Mar 27, 2023
f2b2fda
update to ncarenv/23.03
jedwards4b Mar 31, 2023
d0093a5
Update module versions for intel compiler
sjsprecious Apr 11, 2023
4023d67
Merge branch 'main' into add_gpu_gust
sjsprecious Apr 17, 2023
4e375b2
update esmf version on gust
jedwards4b Apr 18, 2023
7224a37
load ncarenv and ncarcompilers manually
sjsprecious Apr 18, 2023
822f7b7
update the ncarenv and cesmdev module on Gust
sjsprecious Apr 25, 2023
21fc13a
merge to ccs_config_cesm0.0.71
jedwards4b May 25, 2023
aaec12b
fix unresolved merge issue
jedwards4b May 25, 2023
df29563
update default wallclock time
jedwards4b May 26, 2023
823653d
merge jian/add_gpu_gust
jedwards4b May 26, 2023
a002bdc
add GPU_OFFLOAD
jedwards4b May 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions machines/cmake_macros/gust.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ set(NETCDF_PATH "$ENV{NETCDF}")
set(PIO_FILESYSTEM_HINTS "lustre")
set(PNETCDF_PATH "$ENV{PNETCDF}")
# If we want to use cray-libsci instead of mkl uncomment this line as well as the module in config_machines.xml
string(REPLACE "-mkl=cluster" "" SLIBS "${SLIBS}")
#string(REPLACE "-mkl=cluster" "-qmkl=cluster" SLIBS "${SLIBS}")
string(REPLACE "-mkl=cluster" "-qmkl=cluster" SLIBS "${SLIBS}")
#string(APPEND CPPDEFS " -DNO_SHR_VMATH ")
string(APPEND CPPDEFS " -DHAVE_GETTID")
46 changes: 0 additions & 46 deletions machines/cmake_macros/nvhpc-gpu.cmake
View file
Open in desktop

This file was deleted.

15 changes: 0 additions & 15 deletions machines/cmake_macros/nvhpc-gpu_casper.cmake
View file
Open in desktop

This file was deleted.

19 changes: 19 additions & 0 deletions machines/cmake_macros/nvhpc.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,3 +49,22 @@ set(MPIFC "mpif90")
set(SCC "nvc")
set(SCXX "nvc++")
set(SFC "nvfortran")
if (GPU_TYPE STREQUAL v100 AND GPU_OFFLOAD STREQUAL openacc)
string(APPEND GPUFLAGS " -acc -gpu=cc70,lineinfo,nofma -Minfo=accel ")
endif()
if (GPU_TYPE STREQUAL v100 AND GPU_OFFLOAD STREQUAL openmp)
string(APPEND GPUFLAGS " -mp=gpu -gpu=cc70,lineinfo,nofma -Minfo=accel ")
endif()
if (GPU_TYPE STREQUAL v100 AND GPU_OFFLOAD STREQUAL combined)
string(APPEND GPUFLAGS " -acc -gpu=cc70,lineinfo,nofma -mp=gpu -Minfo=accel ")
endif()

if (GPU_TYPE STREQUAL a100 AND GPU_OFFLOAD STREQUAL openacc)
string(APPEND GPUFLAGS " -acc -gpu=cc80,lineinfo,nofma -Minfo=accel ")
endif()
if (GPU_TYPE STREQUAL a100 AND GPU_OFFLOAD STREQUAL openmp)
string(APPEND GPUFLAGS " -mp=gpu -gpu=cc80,lineinfo,nofma -Minfo=accel ")
endif()
if (GPU_TYPE STREQUAL a100 AND GPU_OFFLOAD STREQUAL combined)
string(APPEND GPUFLAGS " -acc -gpu=cc80,lineinfo,nofma -mp=gpu -Minfo=accel")
endif()
1 change: 1 addition & 0 deletions machines/cmake_macros/nvhpc_casper.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@ if (MPILIB STREQUAL mpi-serial)
string(APPEND SLIBS " -ldl")
endif()
string(APPEND SLIBS " -L${NETCDF_PATH}/lib -lnetcdf -lnetcdff")
message("GPU_TYPE is ${GPU_TYPE} GPU_OFFLOAD is ${GPU_OFFLOAD}")
46 changes: 0 additions & 46 deletions machines/cmake_macros/pgi-gpu.cmake
View file
Open in desktop

This file was deleted.

15 changes: 0 additions & 15 deletions machines/cmake_macros/pgi-gpu_casper.cmake
View file
Open in desktop

This file was deleted.

45 changes: 17 additions & 28 deletions machines/config_batch.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,34 +226,14 @@
<!-- casper pbs -->
<batch_system MACH="casper" type="pbs">
<batch_submit>qsub</batch_submit>

<directives queue="casper" compiler="pgi-gpu">
<!-- Turn on MPS server manually -->
<!-- This is a temporary solution and should be removed once MPS is integrated into PBS on Casper -->
<directive default="/bin/bash" > -S /glade/u/apps/dav/opt/nvidia-mps/mps_bash </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=700GB:ngpus={{ ngpus_per_node }} </directive>
<directive> -l gpu_type=v100 </directive>
</directives>

<directives queue="casper" compiler="nvhpc-gpu">
<!-- Turn on MPS server manually -->
<!-- This is a temporary solution and should be removed once MPS is integrated into PBS on Casper -->
<directive default="/bin/bash" > -S /glade/u/apps/dav/opt/nvidia-mps/mps_bash </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=700GB:ngpus={{ ngpus_per_node }} </directive>
<directive> -l gpu_type=v100 </directive>
</directives>

<directives queue="casper" compiler="nvhpc">
<directive default="/bin/bash" > -S {{ shell }} </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=700GB:ngpus={{ ngpus_per_node }} </directive>
</directives>

<directives queue="casper" compiler="pgi">
<submit_args>
<argument> -l gpu_type=$GPU_TYPE </argument>
</submit_args>
<directives queue="casper" compiler="nvhpc" gpu_enabled="true">
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this works but I am not sure how gpu_enabled actually works. Is it an XML variable defined somewhere? And how is it set to True or False during the build. A brief explanation will be very helpful.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is done here: https://github.com/jedwards4b/cime/blob/add_gpu_gust/CIME/case/case.py#L457

gpu_enabled is an attribute of the case object and is set to true if GPU_TYPE is set to a valid value.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jedwards4b for the details. That is very helpful!

<directive default="/bin/bash" > -S {{ shell }} </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=700GB:ngpus={{ ngpus_per_node }} </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=700GB:ngpus={{ ngpus_per_node }}:mps=1 </directive>
</directives>

<directives queue="casper" compiler="intel">
<directives queue="casper" gpu_enabled="false">
<directive default="/bin/bash" > -S {{ shell }} </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=700GB:ngpus={{ ngpus_per_node }} </directive>
</directives>
Expand Down Expand Up @@ -490,12 +470,21 @@

<batch_system MACH="gust" type="pbs" >
<batch_submit>qsub</batch_submit>
<directives>
<submit_args>
<argument> -l gpu_type=$GPU_TYPE </argument>
</submit_args>
<directives queue="main" gpu_enabled="false">
<directive default="/bin/bash" > -S {{ shell }} </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_tasks_per_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}</directive>
</directives>
<directives queue="main" gpu_enabled="true">
<directive default="/bin/bash" > -S {{ shell }} </directive>
<directive> -l select={{ num_nodes }}:ncpus={{ max_cputasks_per_gpu_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=430GB:ngpus={{ ngpus_per_node }} </directive>
<!-- Do not use MPS on Gust yet as it conflicts with cray-mpich library -->
<!-- <directive> -l select={{ num_nodes }}:ncpus={{ max_cputasks_per_gpu_node }}:mpiprocs={{ tasks_per_node }}:ompthreads={{ thread_count }}:mem=430GB:ngpus={{ ngpus_per_node }}:mps=1 </directive> -->
</directives>
<queues>
<queue walltimemax="4:00:00" nodemin="1" nodemax="8" >main</queue>
<queue walltimemax="2:00:00" nodemin="1" nodemax="8" >main</queue>
<queue walltimemax="1:00:00" nodemin="9" nodemax="16" >bigcpu</queue>
</queues>
</batch_system>
Expand Down
76 changes: 22 additions & 54 deletions machines/config_machines.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -410,19 +410,22 @@ This allows using a different mpirun command to launch unit tests
<DESC>NCAR GPU platform, os is Linux, 36 pes/node, batch system is pbs</DESC>
<NODENAME_REGEX>casper*</NODENAME_REGEX>
<OS>LINUX</OS>
<COMPILERS>pgi,intel,nvhpc,pgi-gpu,nvhpc-gpu</COMPILERS>
<COMPILERS>nvhpc,intel</COMPILERS>
<MPILIBS>openmpi</MPILIBS>
<CIME_OUTPUT_ROOT>/glade/scratch/$USER</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/glade/p/cgd/tss/CTSM_datm_forcing_data</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>$ENV{CESMDATAROOT}/cesm_baselines</BASELINE_ROOT>
<CCSM_CPRNC>$ENV{CESMDATAROOT}/tools/cime/tools/cprnc/cprnc</CCSM_CPRNC>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove pgi and *-gpu at line 413.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know I haven't completed the cleanup here yet.

<GMAKE_J>8</GMAKE_J>
<BATCH_SYSTEM>pbs</BATCH_SYSTEM>
<SUPPORTED_BY>ASAP/CISL</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>36</MAX_TASKS_PER_NODE>
<MAX_GPUS_PER_NODE>8</MAX_GPUS_PER_NODE>
<MAX_GPUS_PER_NODE compiler="nvhpc">8</MAX_GPUS_PER_NODE>
<MAX_MPITASKS_PER_NODE>36</MAX_MPITASKS_PER_NODE>
<MAX_CPUTASKS_PER_GPU_NODE>36</MAX_CPUTASKS_PER_GPU_NODE>
<GPU_TYPES>v100,a100</GPU_TYPES>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we specify gpu_offload variable as well? For machine with AMD GPUs, openacc and combined may not be a valid input.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we handle fortran do concurrent here?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great question! In this case, combined may be a confusing word, too. Is there any plan to introduce do concurrent to CAM in the future?

<PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
<mpirun mpilib="default">
<executable>mpirun</executable>
Expand Down Expand Up @@ -450,54 +453,22 @@ This allows using a different mpirun command to launch unit tests
<command name="load">ncarenv/1.3</command>
<command name="load">cmake/3.18.2</command>
</modules>
<modules compiler="pgi">
<command name="load">pgi/20.4</command>
</modules>
<modules compiler="pgi-gpu">
<command name="load">pgi/20.4</command>
</modules>
<modules compiler="nvhpc">
<command name="load">nvhpc/22.2</command>
</modules>
<modules compiler="nvhpc-gpu">
<command name="load">nvhpc/22.2</command>
</modules>
<modules compiler="intel">
<command name="load">intel/19.1.1</command>
<command name="load">mkl/2020.0.1</command>
</modules>
<modules mpilib="openmpi" compiler="pgi">
<command name="load">openmpi/4.1.0</command>
<command name="load">netcdf-mpi/4.8.0</command>
<command name="load">pnetcdf/1.12.2</command>
</modules>
<modules mpilib="mpi-serial" compiler="pgi">
<command name="load">netcdf/4.8.0</command>
</modules>
<modules mpilib="openmpi" compiler="pgi-gpu">
<command name="load">openmpi/4.1.0</command>
<command name="load">netcdf-mpi/4.7.4</command>
<command name="load">pnetcdf/1.12.2</command>
<command name="load">cuda/11.0.3</command>
</modules>
<modules mpilib="mpi-serial" compiler="pgi-gpu">
<command name="load">netcdf/4.7.4</command>
</modules>
<modules mpilib="openmpi" compiler="nvhpc">
<command name="load">openmpi/4.1.4</command>
<command name="load">netcdf-mpi/4.8.1</command>
<command name="load">pnetcdf/1.12.3</command>
</modules>
<modules mpilib="mpi-serial" compiler="nvhpc">
<command name="load">netcdf/4.8.1</command>
</modules>
<modules mpilib="openmpi" compiler="nvhpc-gpu">
<command name="load">openmpi/4.1.4</command>
<command name="load">netcdf-mpi/4.8.1</command>
<command name="load">pnetcdf/1.12.3</command>
<command name="load">cuda/11.4.0</command>
<modules gpu_type="!none">
<command name="load">cuda/11.6</command>
</modules>
<modules mpilib="mpi-serial" compiler="nvhpc-gpu">
<modules mpilib="mpi-serial" compiler="nvhpc">
<command name="load">netcdf/4.8.1</command>
</modules>
<modules mpilib="openmpi" compiler="intel">
Expand All @@ -517,29 +488,21 @@ This allows using a different mpirun command to launch unit tests
<command name="use">/glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/intel/19.1.1/</command>
<command name="load">esmf-8.4.0b08_casper-ncdfio-openmpi-O</command>
</modules>
<modules compiler="nvhpc-gpu" mpilib="openmpi" DEBUG="TRUE">
<modules compiler="nvhpc" mpilib="openmpi" DEBUG="TRUE">
<command name="use">/glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/nvhpc/22.2/</command>
<command name="load">esmf-8.4.1b01-ncdfio-openmpi-g</command>
<command name="load">esmf-8.4.1_casper-ncdfio-openmpi-g</command>
</modules>
<modules compiler="nvhpc-gpu" mpilib="openmpi" DEBUG="FALSE">
<modules compiler="nvhpc" mpilib="openmpi" DEBUG="FALSE">
<command name="use">/glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/nvhpc/22.2/</command>
<command name="load">esmf-8.4.1b01-ncdfio-openmpi-O</command>
</modules>
<modules compiler="pgi" mpilib="openmpi" DEBUG="TRUE">
<command name="use">/glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/pgi/20.4/</command>
<command name="load">esmf-8.4.0b08_casper-ncdfio-openmpi-g</command>
</modules>
<modules compiler="pgi" mpilib="openmpi" DEBUG="FALSE">
<command name="use">/glade/p/cesmdata/cseg/PROGS/modulefiles/esmfpkgs/pgi/20.4/</command>
<command name="load">esmf-8.2.0b11_casper-ncdfio-openmpi-O</command>
<command name="load">esmf-8.4.1_casper-ncdfio-openmpi-O</command>
</modules>
<modules>
<command name="load">ncarcompilers/0.5.0</command>
</modules>
<modules compiler="!pgi" DEBUG="FALSE" mpilib="openmpi">
<modules DEBUG="FALSE" mpilib="openmpi">
<command name="load">pio/2.5.10</command>
</modules>
<modules compiler="!pgi" DEBUG="TRUE" mpilib="openmpi">
<modules DEBUG="TRUE" mpilib="openmpi">
<command name="load">pio/2.5.10d</command>
</modules>
</module_system>
Expand Down Expand Up @@ -580,7 +543,7 @@ This allows using a different mpirun command to launch unit tests
<DIN_LOC_ROOT_CLMFORC>/glade/p/cgd/tss/CTSM_datm_forcing_data</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>$ENV{CESMDATAROOT}/cesm_baselines</BASELINE_ROOT>
<CCSM_CPRNC>$ENV{CESMDATAROOT}/tools/cime/tools/cprnc/cprnc.cheyenne</CCSM_CPRNC>
<CCSM_CPRNC>$ENV{CESMDATAROOT}/tools/cime/tools/cprnc/cprnc</CCSM_CPRNC>
<GMAKE_J>8</GMAKE_J>
<BATCH_SYSTEM>pbs</BATCH_SYSTEM>
<SUPPORTED_BY>cseg</SUPPORTED_BY>
Expand Down Expand Up @@ -1850,7 +1813,11 @@ This allows using a different mpirun command to launch unit tests
<BATCH_SYSTEM>pbs</BATCH_SYSTEM>
<SUPPORTED_BY>cseg</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>128</MAX_TASKS_PER_NODE>
<MAX_GPUS_PER_NODE>4</MAX_GPUS_PER_NODE>
<MAX_MPITASKS_PER_NODE>128</MAX_MPITASKS_PER_NODE>
<MAX_CPUTASKS_PER_GPU_NODE>64</MAX_CPUTASKS_PER_GPU_NODE>
<GPU_TYPES>a100</GPU_TYPES>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same question about gpu_offload variable.

<GPU_OFFLOAD>openacc,openmp,combined</GPU_OFFLOAD>
<PROJECT_REQUIRED>TRUE</PROJECT_REQUIRED>
<mpirun mpilib="default">
<executable>mpiexec</executable>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At line 1838, shall we do purge first before loading any module?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a feature of modules on gust (and soon derecho) - the two modules loaded above the purge are sticky and not affected by the purge command.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification. That is clear to me now.

Expand Down Expand Up @@ -1907,11 +1874,12 @@ This allows using a different mpirun command to launch unit tests
<modules mpilib="mpi-serial">
<command name="load">mpi-serial/2.3.0</command>
</modules>

<modules gpu_type="!none">
<command name="load">cuda/11.7.1</command>
</modules>
<modules mpilib="mpi-serial">
<command name="load">netcdf/4.9.1</command>
</modules>

<modules mpilib="!mpi-serial">
<command name="load">netcdf-mpi/4.9.1</command>
<command name="load">parallel-netcdf/1.12.3</command>
Expand Down
6 changes: 0 additions & 6 deletions machines/mpi_run_gpu.casper
View file
Open in desktop

This file was deleted.