Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge in DoD onyx mods #3100

Closed
wants to merge 7 commits into from
Closed

Merge in DoD onyx mods #3100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e725c80
port to onyx
apcraig Jan 18, 2019
6a2d2ea
update mpascice graph partition files and ARRM60to10 default pe layout
apcraig Jan 31, 2019
b6d0120
update pe layout for oARRM60to6
apcraig May 25, 2019
1271f4b
Merge commit '8a8eefb5daa7caa5a69b4c913a' into apcraig/onyx1
apcraig Jun 21, 2019
9443fcf
add TL319_oARRM pe layout
apcraig Jul 25, 2019
f956fd5
update to milena/e3sm-arctic-2019
apcraig Jul 30, 2019
bebf06d
update mpaso namelist defaults for ARRM configurations
apcraig Jul 31, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 74 additions & 0 deletions cime/config/e3sm/allactive/config_pesall.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8446,4 +8446,78 @@
</pes>
</mach>
</grid>
<grid name="a%T62.+oi%oARRM60to10|a%TL319.+oi%oARRM60to10">
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest to add these changes for a specific machine. Also, @apcraig: we should talk about the improvements you experienced with these settings (we can do this by email).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. I will take care of that.

<mach name="any">
<pes compset=".*MPASCICE.+MPASO.+" pesize="any">
<comment>none</comment>
<ntasks>
<ntasks_atm>64</ntasks_atm>
<ntasks_lnd>64</ntasks_lnd>
<ntasks_rof>64</ntasks_rof>
<ntasks_ice>800</ntasks_ice>
<ntasks_ocn>2400</ntasks_ocn>
<ntasks_glc>64</ntasks_glc>
<ntasks_wav>64</ntasks_wav>
<ntasks_cpl>800</ntasks_cpl>
</ntasks>
<nthrds>
<nthrds_atm>1</nthrds_atm>
<nthrds_lnd>1</nthrds_lnd>
<nthrds_rof>1</nthrds_rof>
<nthrds_ice>1</nthrds_ice>
<nthrds_ocn>1</nthrds_ocn>
<nthrds_glc>1</nthrds_glc>
<nthrds_wav>1</nthrds_wav>
<nthrds_cpl>1</nthrds_cpl>
</nthrds>
<rootpe>
<rootpe_atm>0</rootpe_atm>
<rootpe_lnd>0</rootpe_lnd>
<rootpe_rof>0</rootpe_rof>
<rootpe_ice>0</rootpe_ice>
<rootpe_ocn>800</rootpe_ocn>
<rootpe_glc>0</rootpe_glc>
<rootpe_wav>0</rootpe_wav>
<rootpe_cpl>0</rootpe_cpl>
</rootpe>
</pes>
</mach>
</grid>
<grid name="a%T62.+oi%oARRM60to6|a%TL319.+oi%oARRM60to6">
<mach name="any">
<pes compset=".*MPASCICE.+MPASO.+" pesize="any">
<comment>none</comment>
<ntasks>
<ntasks_atm>64</ntasks_atm>
<ntasks_lnd>64</ntasks_lnd>
<ntasks_rof>64</ntasks_rof>
<ntasks_ice>4096</ntasks_ice>
<ntasks_ocn>8192</ntasks_ocn>
<ntasks_glc>64</ntasks_glc>
<ntasks_wav>64</ntasks_wav>
<ntasks_cpl>4096</ntasks_cpl>
</ntasks>
<nthrds>
<nthrds_atm>1</nthrds_atm>
<nthrds_lnd>1</nthrds_lnd>
<nthrds_rof>1</nthrds_rof>
<nthrds_ice>1</nthrds_ice>
<nthrds_ocn>1</nthrds_ocn>
<nthrds_glc>1</nthrds_glc>
<nthrds_wav>1</nthrds_wav>
<nthrds_cpl>1</nthrds_cpl>
</nthrds>
<rootpe>
<rootpe_atm>0</rootpe_atm>
<rootpe_lnd>0</rootpe_lnd>
<rootpe_rof>0</rootpe_rof>
<rootpe_ice>0</rootpe_ice>
<rootpe_ocn>4096</rootpe_ocn>
<rootpe_glc>0</rootpe_glc>
<rootpe_wav>0</rootpe_wav>
<rootpe_cpl>0</rootpe_cpl>
</rootpe>
</pes>
</mach>
</grid>
</config_pes>
13 changes: 13 additions & 0 deletions cime/config/e3sm/machines/config_batch.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,19 @@
</queues>
</batch_system>

<!-- onyx is PBS -->
<batch_system MACH="onyx" type="pbs" >
<directives>
<directive>-A {{ PROJECT }}</directive>
<directive>-l application=Regional-Arctic-System-Model</directive>
<directive>-l select={{ num_nodes }}:ncpus={{ MAX_MPITASKS_PER_NODE }}:mpiprocs={{ tasks_per_node }}</directive>
</directives>
<queues>
<queue walltimemax="00:59:00" nodemax="155" strict="true">debug</queue>
<queue walltimemax="04:00:00" default="true">frontier</queue>
</queues>
</batch_system>

<batch_system MACH="bebop" type="slurm" >
<queues>
<queue walltimemax="00:30:00" nodemax="64" strict="true">debug</queue>
Expand Down
15 changes: 15 additions & 0 deletions cime/config/e3sm/machines/config_compilers.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1661,6 +1661,21 @@ ntel/x86_64/2013/composer_xe_2013/composer_xe_2013_sp1.3.174/mkl/include </base>
</SLIBS>
</compiler>

<compiler MACH="onyx" COMPILER="intel">
<CONFIG_ARGS>
<base> --host=Linux </base>
</CONFIG_ARGS>
<FFLAGS>
<base> -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model consistent -fimf-use-svml </base>
<append DEBUG="FALSE"> -O2 -debug minimal -qno-opt-dynamic-align </append>
</FFLAGS>
<PETSC_PATH>$ENV{PETSC_DIR}</PETSC_PATH>
<SLIBS>
<append> -Wl,--as-needed,-L$ENV{NETCDF_DIR}/lib -lnetcdf </append>
<append> -mkl -lpthread </append>
</SLIBS>
</compiler>

<compiler MACH="sandiatoss3" COMPILER="intel">
<ALBANY_PATH>/projects/ccsm/AlbanyTrilinos_06262017/Albany/build/install</ALBANY_PATH>
<CFLAGS>
Expand Down
107 changes: 107 additions & 0 deletions cime/config/e3sm/machines/config_machines.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,113 @@

</machine>

<machine MACH="onyx">
<DESC>ERDC XC40, os is CNL, 44 pes/node, batch system is PBS</DESC>
<NODENAME_REGEX>onyx</NODENAME_REGEX>
<OS>CNL</OS>
<COMPILERS>intel</COMPILERS>
<MPILIBS>mpich</MPILIBS>
<CHARGE_ACCOUNT>NPSCA07935YF5</CHARGE_ACCOUNT>
<SAVE_TIMING_DIR>/p/app/unsupported/RASM/acme</SAVE_TIMING_DIR>
<SAVE_TIMING_DIR_PROJECTS>.*</SAVE_TIMING_DIR_PROJECTS>
<CIME_OUTPUT_ROOT>$ENV{WORKDIR}</CIME_OUTPUT_ROOT>
<DIN_LOC_ROOT>/p/app/unsupported/RASM/acme/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/p/app/unsupported/RASM/acme/inputdata/atm/datm7</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>
<BASELINE_ROOT>/p/app/unsupported/RASM/acme/baselines/$COMPILER</BASELINE_ROOT>
<CCSM_CPRNC>/p/app/unsupported/RASM/tools/cprnc/cprnc</CCSM_CPRNC>
<GMAKE_J>8</GMAKE_J>
<TESTS>e3sm_developer</TESTS>
<BATCH_SYSTEM>pbs</BATCH_SYSTEM>
<SUPPORTED_BY>e3sm</SUPPORTED_BY>
<MAX_TASKS_PER_NODE>44</MAX_TASKS_PER_NODE>
<MAX_MPITASKS_PER_NODE>44</MAX_MPITASKS_PER_NODE>
<mpirun mpilib="default">
<executable>aprun</executable>
<arguments>
<arg name="num_tasks">-n {{ total_tasks }}</arg>
<arg name="tasks_per_node">-N $SHELL{if [ `./xmlquery --value MAX_MPITASKS_PER_NODE` -gt `./xmlquery --value TOTAL_TASKS` ];then echo `./xmlquery --value TOTAL_TASKS`;else echo `./xmlquery --value MAX_MPITASKS_PER_NODE`;fi;}</arg>
<arg name="hyperthreading">--cc depth -d $SHELL{echo `./xmlquery --value MAX_TASKS_PER_NODE`/`./xmlquery --value MAX_MPITASKS_PER_NODE`|bc} -j $SHELL{if [ 64 -ge `./xmlquery --value MAX_TASKS_PER_NODE` ];then echo 1;else echo `./xmlquery --value MAX_TASKS_PER_NODE`/64|bc;fi;}</arg>
<arg name="env_vars">$ENV{SMP_VARS} $ENV{labeling}</arg>
</arguments>
</mpirun>
<module_system type="module">
<init_path lang="perl">/opt/modules/default/init/perl.pm</init_path>
<init_path lang="python">/opt/modules/default/init/python.py</init_path>
<init_path lang="sh">/opt/modules/default/init/sh</init_path>
<init_path lang="csh">/opt/modules/default/init/csh</init_path>
<cmd_path lang="perl">/opt/modules/default/bin/modulecmd perl</cmd_path>
<cmd_path lang="python">/opt/modules/default/bin/modulecmd python</cmd_path>
<cmd_path lang="sh">module</cmd_path>
<cmd_path lang="csh">module</cmd_path>
<modules>
<command name="rm">PrgEnv-intel</command>
<command name="rm">PrgEnv-cray</command>
<command name="rm">PrgEnv-gnu</command>
<command name="rm">PrgEnv-pgi</command>
<command name="rm">intel</command>
<command name="rm">cce</command>
<command name="rm">gcc</command>
<command name="rm">cray-parallel-netcdf</command>
<command name="rm">cray-parallel-hdf5</command>
<command name="rm">pmi</command>
<command name="rm">cray-libsci</command>
<command name="rm">cray-mpich2</command>
<command name="rm">cray-mpich</command>
<command name="rm">cray-netcdf</command>
<command name="rm">cray-hdf5</command>
<command name="rm">cray-netcdf-hdf5parallel</command>
<command name="rm">craype-sandybridge</command>
<command name="rm">craype-ivybridge</command>
<command name="rm">craype</command>
<command name="rm">papi</command>
<command name="rm">cray-petsc</command>
<command name="rm">cray-libsci</command>
<command name="rm">esmf</command>
<command name="rm">craype</command>
</modules>

<modules compiler="intel">
<command name="load">PrgEnv-intel/6.0.3</command>
<command name="rm">intel</command>
<command name="load">intel/17.0.1.132</command>
<command name="rm">cray-mpich</command>
<command name="load">cray-mpich/7.5.2</command>
<command name="rm">cray-hdf5</command>
<command name="rm">cray-hdf5-parallel</command>
<command name="rm">cray-netcdf-hdf5parallel</command>
<command name="rm">cray-parallel-netcdf</command>
<command name="rm">netcdf</command>
<command name="load">cray-netcdf/4.4.1.1</command>
<command name="load">cray-hdf5/1.10.0.1</command>
<command name="load">cray-parallel-netcdf/1.7.0</command>
<command name="rm">cray-libsci</command>
</modules>
</module_system>
<RUNDIR>$CIME_OUTPUT_ROOT/$CASE/run</RUNDIR>
<EXEROOT>$CIME_OUTPUT_ROOT/$CASE/bld</EXEROOT>
<TEST_TPUT_TOLERANCE>0.1</TEST_TPUT_TOLERANCE>
<environment_variables>
<env name="MPICH_ENV_DISPLAY">1</env>
<env name="MPICH_VERSION_DISPLAY">1</env>
<env name="NETCDF_DIR">/opt/cray/pe/netcdf/4.4.1.1/intel/15.0</env>
<!--env name="MPICH_CPUMASK_DISPLAY">1</env-->

<env name="OMP_STACKSIZE">64M</env>
<env name="OMP_PROC_BIND">spread</env>
<env name="OMP_PLACES">threads</env>

<!--env name="HDF5_DISABLE_VERSION_CHECK">2</env-->
<env name="HDF5_USE_FILE_LOCKING">FALSE</env>
</environment_variables>
<environment_variables compiler="intel">
<env name="FORT_BUFFERED">yes</env>
</environment_variables>
<environment_variables compiler="intel18">
<env name="FORT_BUFFERED">yes</env>
</environment_variables>
</machine>

<machine MACH="cori-haswell">
<DESC>Cori. XC40 Cray system at NERSC. Haswell partition. os is CNL, 32 pes/node, batch system is SLURM</DESC>
<NODENAME_REGEX>cori-knl-is-default</NODENAME_REGEX>
Expand Down
1 change: 1 addition & 0 deletions cime/config/e3sm/machines/config_pio.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,7 @@
<entry id="CPL_PIO_TYPENAME">
<values>
<value mach="mira" grid="a%ne120">netcdf</value>
<value mach="onyx" >netcdf</value>
</values>
</entry>

Expand Down
14 changes: 12 additions & 2 deletions components/mpas-ocean/bld/namelist_files/namelist_defaults_mpaso.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,15 @@
<config_use_standardGM ocn_grid="oRRS18to6">.false.</config_use_standardGM>
<config_use_standardGM ocn_grid="oRRS18to6v3">.false.</config_use_standardGM>
<config_use_standardGM ocn_grid="oRRS15to5">.false.</config_use_standardGM>
<config_use_standardGM ocn_grid="oARRM60to10">.false.</config_use_standardGM>
<config_use_standardGM ocn_grid="oARRM60to6">.false.</config_use_standardGM>
<config_use_standardGM ocn_grid="oARRM60to10">.true.</config_use_standardGM>
<config_use_standardGM ocn_grid="oARRM60to6">.true.</config_use_standardGM>
<config_use_Redi_surface_layer_tapering>.false.</config_use_Redi_surface_layer_tapering>
<config_Redi_surface_layer_tapering_extent>0.0</config_Redi_surface_layer_tapering_extent>
<config_use_Redi_bottom_layer_tapering>.false.</config_use_Redi_bottom_layer_tapering>
<config_Redi_bottom_layer_tapering_depth>0.0</config_Redi_bottom_layer_tapering_depth>
<config_GM_Bolus_kappa_function>'constant'</config_GM_Bolus_kappa_function>
<config_GM_Bolus_kappa_function ocn_grid="oARRM60to10">'ramp'</config_GM_Bolus_kappa_function>
<config_GM_Bolus_kappa_function ocn_grid="oARRM60to6">'ramp'</config_GM_Bolus_kappa_function>
<config_standardGM_tracer_kappa>1800.0</config_standardGM_tracer_kappa>
<config_standardGM_tracer_kappa ocn_grid="oEC60to30v3wLI" ocn_forcing="datm_forced_restoring" >600.0</config_standardGM_tracer_kappa>
<config_GM_Bolus_kappa_min>0.0</config_GM_Bolus_kappa_min>
Expand Down Expand Up @@ -440,14 +442,20 @@
<config_use_activeTracers>.true.</config_use_activeTracers>
<config_use_activeTracers_surface_bulk_forcing>.true.</config_use_activeTracers_surface_bulk_forcing>
<config_use_activeTracers_surface_restoring>.false.</config_use_activeTracers_surface_restoring>
<config_use_activeTracers_surface_restoring ocn_grid="oARRM60to10">.true.</config_use_activeTracers_surface_restoring>
<config_use_activeTracers_surface_restoring ocn_grid="oARRM60to6">.true.</config_use_activeTracers_surface_restoring>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these should not be default because we only do restoring in G-cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will back out all of the changes to the mpaso namelist_defaults. I think that is more consistent with the overall strategy of using user_nl_mpaso to set things up and the benefit is the namelist defaults file remains consistent with master.

<config_use_activeTracers_interior_restoring>.false.</config_use_activeTracers_interior_restoring>
<config_use_activeTracers_exponential_decay>.false.</config_use_activeTracers_exponential_decay>
<config_use_activeTracers_idealAge_forcing>.false.</config_use_activeTracers_idealAge_forcing>
<config_use_activeTracers_ttd_forcing>.false.</config_use_activeTracers_ttd_forcing>
<config_use_surface_salinity_monthly_restoring>.false.</config_use_surface_salinity_monthly_restoring>
<config_use_surface_salinity_monthly_restoring ocn_grid="oARRM60to10">.true.</config_use_surface_salinity_monthly_restoring>
<config_use_surface_salinity_monthly_restoring ocn_grid="oARRM60to6">.true.</config_use_surface_salinity_monthly_restoring>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above.

<config_surface_salinity_monthly_restoring_compute_interval>'0000-00-01_00:00:00'</config_surface_salinity_monthly_restoring_compute_interval>
<config_salinity_restoring_constant_piston_velocity>1.585e-6</config_salinity_restoring_constant_piston_velocity>
<config_salinity_restoring_max_difference>0.5</config_salinity_restoring_max_difference>
<config_salinity_restoring_max_difference ocn_grid="oARRM60to10">100.</config_salinity_restoring_max_difference>
<config_salinity_restoring_max_difference ocn_grid="oARRM60to6">100.</config_salinity_restoring_max_difference>
<config_salinity_restoring_under_sea_ice>.false.</config_salinity_restoring_under_sea_ice>

<!-- tracer_forcing_debugTracers -->
Expand Down Expand Up @@ -817,6 +825,8 @@

<!-- AM_eddyProductVariables -->
<config_AM_eddyProductVariables_enable>.false.</config_AM_eddyProductVariables_enable>
<config_AM_eddyProductVariables_enable ocn_grid="oARRM60to10">.true.</config_AM_eddyProductVariables_enable>
<config_AM_eddyProductVariables_enable ocn_grid="oARRM60to6">.true.</config_AM_eddyProductVariables_enable>
<config_AM_eddyProductVariables_enable ocn_grid="oRRS30to10v3">.true.</config_AM_eddyProductVariables_enable>
<config_AM_eddyProductVariables_enable ocn_grid="oRRS30to10v3wLI">.true.</config_AM_eddyProductVariables_enable>
<config_AM_eddyProductVariables_enable ocn_grid="oRRS18to6v3">.true.</config_AM_eddyProductVariables_enable>
Expand Down
3 changes: 2 additions & 1 deletion components/mpas-seaice/bld/build-namelist-section
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,8 @@ add_default($nl, 'config_full_abort_write');
# Namelist group: decomposition #
#################################

add_default($nl, 'config_block_decomp_file_prefix', 'val'=>"'${DIN_LOC_ROOT}/ice/mpas-cice/${ICE_GRID}/mpas-cice.graph.info.${date_stamp}.part.'");
#add_default($nl, 'config_block_decomp_file_prefix', 'val'=>"'${DIN_LOC_ROOT}/ice/mpas-cice/${ICE_GRID}/mpas-cice.graph.info.${date_stamp}.part.'");
add_default($nl, 'config_block_decomp_file_prefix', 'val'=>"'${DIN_LOC_ROOT}/ice/mpas-cice/${ICE_GRID}/mpas-seaice.graph.v2.${date_stamp}.part.'");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please explain this change? Shouldn't the default be the graph.info file since we do not know for sure if the v2 version is available for all meshes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed this and will remove it. I just grepped for graph in the mpas-seaice stuff and made changes in all places that seemed appropriate. I have now looked at the implementation and agree this should be left unchanged.

add_default($nl, 'config_number_of_blocks');
add_default($nl, 'config_explicit_proc_decomp');
add_default($nl, 'config_proc_decomp_file_prefix');
Expand Down
2 changes: 0 additions & 2 deletions components/mpas-seaice/cime_config/buildnml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,8 +175,6 @@ def buildnml(case, caseroot, compname):
elif ice_grid == 'oARRM60to10':
grid_date += '180715'
grid_prefix += 'seaice.ARRM60to10'
#decomp_date += '180723'
#decomp_prefix += 'mpas-cice.graph.info.'
decomp_date += '190129'
decomp_prefix += 'mpas-seaice.graph.v2.'
elif ice_grid == 'oARRM60to6':
Expand Down