Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jgfouca/ccsm utils/redsky config #16

Merged
merged 9 commits into from
Sep 18, 2014
Merged

Jgfouca/ccsm utils/redsky config #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6326312
Progress getting ACME to run on redsky
jgfouca Aug 21, 2014
b84a7b7
ccsm_utils: Add redsky prebeta tests to testlist
jgfouca Aug 28, 2014
49760a9
update CESMSCRATCHROOT for redsky
Sep 3, 2014
9a8022b
Switch redsky filesystem to gscratch1, it's faster than gscratch2
jgfouca Sep 9, 2014
0340fa7
Changes to redsky machine configuration
jgfouca Sep 11, 2014
a612914
Fix to mkbatch.redsky, remove obsolete numa_wrapper stuff
jgfouca Sep 15, 2014
41b6759
Increase time limit of queued jobs for redsky
jgfouca Sep 15, 2014
f915955
Make sure to submit redsky jobs to acceptable partition
jgfouca Sep 15, 2014
5234c85
Add env_mach_specific file for redsky
jgfouca Sep 16, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions scripts/ccsm_utils/Machines/config_compilers.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,13 @@ for mct, etc.
<SUPPORTS_CXX>TRUE</SUPPORTS_CXX>
</compiler>

<compiler COMPILER="intel" MACH="redsky">
<NETCDF_PATH>$(NETCDFROOT)</NETCDF_PATH>
<PNETCDF_PATH></PNETCDF_PATH>
<ADD_SLIBS> $(shell $(NETCDF_PATH)/bin/nc-config --flibs) </ADD_SLIBS>
<ADD_CPPFLAGS> -DHAVE_COMM_F2C </ADD_CPPFLAGS>
</compiler>

<compiler COMPILER="intel14">
<!-- http://software.intel.com/en-us/articles/intel-composer-xe/ -->
<ADD_CPPDEFS> -DFORTRANUNDERSCORE -DNO_R16</ADD_CPPDEFS>
Expand Down
61 changes: 42 additions & 19 deletions scripts/ccsm_utils/Machines/config_machines.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,33 @@

<config_machines>

<!--
<!--

===============================================================
COMPILER and COMPILERS
COMPILER and COMPILERS
===============================================================
If a machine supports multiple compilers - then
If a machine supports multiple compilers - then
- the settings for COMPILERS should reflect the supported compilers
as a comma separated string
- the setting for COMPILER should be the default compiler
- the setting for COMPILER should be the default compiler
(which is one of the values in COMPILERS)

===============================================================
MPILIB and MPILIBS
===============================================================
If a machine supports only one MPILIB is supported - then
the setting for MPILIB and MPILIBS should be blank ("")
If a machine supports only one MPILIB is supported - then
the setting for MPILIB and MPILIBS should be blank ("")
If a machine supports multiple mpi libraries (e.g. mpich and openmpi)
- the settings for MPILIBS should reflect the supported mpi libraries
as a comma separated string

The default settings for COMPILERS and MPILIBS is blank (in config_machines.xml)
The default settings for COMPILERS and MPILIBS is blank (in config_machines.xml)

Normally variable substitutions are not made until the case scripts are run, however variables
of the form $ENV{VARIABLE_NAME} are substituted in create_newcase from the environment
of the form $ENV{VARIABLE_NAME} are substituted in create_newcase from the environment
variable of the same name if it exists.

NOTE: The variable CESMSCRATCHROOT must be fully resolved before a case is built so cannot
NOTE: The variable CESMSCRATCHROOT must be fully resolved before a case is built so cannot
contain other variables defined in cesm.

-->
Expand All @@ -47,13 +47,36 @@
<DOUT_L_MSROOT>USERDEFINED_optional_run</DOUT_L_MSROOT> <!-- complete path to a long term archiving directory -->
<CCSM_BASELINE>USERDEFINED_optional_run</CCSM_BASELINE> <!-- where the cesm testing scripts write and read baseline results -->
<CCSM_CPRNC>USERDEFINED_optional_test</CCSM_CPRNC> <!-- path to the cprnc tool used to compare netcdf history files in testing -->
<BATCHQUERY>USERDEFINED_optional_run</BATCHQUERY>
<BATCHQUERY>USERDEFINED_optional_run</BATCHQUERY>
<BATCHSUBMIT>USERDEFINED_optional_run</BATCHSUBMIT>
<SUPPORTED_BY>USERDEFINED_optional</SUPPORTED_BY>
<SUPPORTED_BY>USERDEFINED_optional</SUPPORTED_BY>
<GMAKE_J>1</GMAKE_J>
<MAX_TASKS_PER_NODE>USERDEFINED_required_build</MAX_TASKS_PER_NODE>
</machine>

<machine MACH="redsky">
<DESC>SNL Redsky cluster, Linux (x86_64)</DESC> <!-- can be anything -->
<OS>LINUX</OS> <!-- LINUX,Darwin,CNL,AIX,BGL,BGP -->
<PIO_CONFIG_OPTS> --enable-filesystem-hints=lustre </PIO_CONFIG_OPTS>
<COMPILERS>intel</COMPILERS> <!-- intel,ibm,pgi,pathscale,gnu,cray,lahey -->
<MPILIBS>openmpi,mpi-serial</MPILIBS> <!-- openmpi, mpich, ibm, mpi-serial -->
<CESMSCRATCHROOT>/gscratch1/$USER/acme_scratch</CESMSCRATCHROOT>
<RUNDIR>$CESMSCRATCHROOT/$CASE/run</RUNDIR>
<EXEROOT>$CESMSCRATCHROOT/$CASE/bld</EXEROOT>
<DIN_LOC_ROOT>/projects/ccsm/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/glade/proj2/cgd/tss</DIN_LOC_ROOT_CLMFORC>
<DOUT_S>FALSE</DOUT_S> <!-- logical for short term archiving -->
<DOUT_S_ROOT>USERDEFINED_optional_run</DOUT_S_ROOT> <!-- complete path to a short term archiving directory -->
<DOUT_L_MSROOT>USERDEFINED_optional_run</DOUT_L_MSROOT> <!-- complete path to a long term archiving directory -->
<CCSM_BASELINE>/projects/ccsm/ccsm_baselines</CCSM_BASELINE>
<CCSM_CPRNC>USERDEFINED_optional_test</CCSM_CPRNC> <!-- path to the cprnc tool used to compare netcdf history files in testing -->
<BATCHQUERY>qstat</BATCHQUERY>
<BATCHSUBMIT>sbatch &lt;</BATCHSUBMIT>
<SUPPORTED_BY>USERDEFINED_optional</SUPPORTED_BY>
<GMAKE_J>4</GMAKE_J>
<MAX_TASKS_PER_NODE>8 </MAX_TASKS_PER_NODE>
</machine>

<machine MACH="bluewaters">
<DESC>ORNL XE6, os is CNL, 32 pes/node, batch system is PBS</DESC>
<COMPILERS>pgi,cray,gnu</COMPILERS>
Expand Down Expand Up @@ -514,7 +537,7 @@
<DIN_LOC_ROOT>/scratch/projects/xsede/CESM/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>/scratch/projects/xsede/CESM/inputdata/lmwg</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$WORK/archive/$CASE</DOUT_S_ROOT>
<DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
<DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
<CCSM_BASELINE>/scratch/projects/xsede/CESM/ccsm_baselines</CCSM_BASELINE>
<CCSM_CPRNC>/scratch/projects/xsede/CESM/tools/cprnc/cprnc</CCSM_CPRNC>
<BATCHQUERY>squeue</BATCHQUERY>
Expand Down Expand Up @@ -558,10 +581,10 @@
<DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>$ENV{CESMROOT}/lmwg</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>$CESMSCRATCHROOT/archive/$CASE</DOUT_S_ROOT>
<DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
<CCSM_BASELINE>$ENV{CESMDATAROOT}/ccsm_baselines</CCSM_BASELINE>
<DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
<CCSM_BASELINE>$ENV{CESMDATAROOT}/ccsm_baselines</CCSM_BASELINE>
<CCSM_CPRNC>$ENV{CESMDATAROOT}/tools/cprnc/cprnc</CCSM_CPRNC>
<BATCHQUERY>bjobs -w</BATCHQUERY>
<BATCHQUERY>bjobs -w</BATCHQUERY>
<BATCHSUBMIT>bsub &lt;</BATCHSUBMIT>
<SUPPORTED_BY>cseg</SUPPORTED_BY>
<GMAKE_J>8</GMAKE_J>
Expand All @@ -580,10 +603,10 @@
<DIN_LOC_ROOT>$ENV{CESMDATAROOT}/inputdata</DIN_LOC_ROOT>
<DIN_LOC_ROOT_CLMFORC>$ENV{CESMROOT}/lmwg</DIN_LOC_ROOT_CLMFORC>
<DOUT_S_ROOT>/glade/scratch/$CCSMUSER/archive/$CASE</DOUT_S_ROOT>
<DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
<CCSM_BASELINE>$ENV{CESMDATAROOT}/ccsm_baselines</CCSM_BASELINE>
<DOUT_L_MSROOT>csm/$CASE</DOUT_L_MSROOT>
<CCSM_BASELINE>$ENV{CESMDATAROOT}/ccsm_baselines</CCSM_BASELINE>
<CCSM_CPRNC>$ENV{CESMDATAROOT}/tools/cprnc/cprnc</CCSM_CPRNC>
<BATCHQUERY>bjobs -w</BATCHQUERY>
<BATCHQUERY>bjobs -w</BATCHQUERY>
<BATCHSUBMIT>bsub &lt;</BATCHSUBMIT>
<SUPPORTED_BY>srinathv -at- ucar.edu</SUPPORTED_BY>
<GMAKE_J>8</GMAKE_J>
Expand Down
39 changes: 39 additions & 0 deletions scripts/ccsm_utils/Machines/env_mach_specific.redsky
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#! /bin/csh -f

#===============================================================================
# Redsky machine specific settings
#===============================================================================

#-------------------------------------------------------------------------------
# Modules
#-------------------------------------------------------------------------------

if (-e /usr/share/Modules/init/csh) then
source /usr/share/Modules/init/csh
module purge

if ($COMPILER == "intel") then
module load intel/intel-14.0.3.174
module load openmpi-intel/1.6
endif

# if ( $MPILIB == "mpi-serial") then
# module load cray-netcdf/4.3.0
# else
# module load cray-netcdf-hdf5parallel/4.3.0
# module load cray-parallel-netcdf/1.3.1.1
# endif

endif

# Get newer cmake
setenv PATH /projects/ccsm/cmake-2.8.10.2-Linux-i386/bin:$PATH

# Get newer netcdf
setenv NETCDFROOT /projects/ccsm/netcdf4.3.2-intel12
setenv PATH $NETCDFROOT/bin:$PATH
setenv LD_LIBRARY_PATH $NETCDFROOT/lib:$LD_LIBRARY_PATH
setenv NETCDF_INCLUDES $NETCDFROOT/include
setenv NETCDF_LIBS $NETCDFROOT/lib


106 changes: 106 additions & 0 deletions scripts/ccsm_utils/Machines/mkbatch.redsky
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
#! /bin/csh -f

#################################################################################
if ($PHASE == set_batch) then
#################################################################################

source ./Tools/ccsm_getenv || exit -1

set ntasks = `${CASEROOT}/Tools/taskmaker.pl -sumonly`
set maxthrds = `${CASEROOT}/Tools/taskmaker.pl -maxthrds`
@ nodes = $ntasks / ${MAX_TASKS_PER_NODE}
if ( $ntasks % ${MAX_TASKS_PER_NODE} > 0) then
@ nodes = $nodes + 1
@ ntasks = $nodes * ${MAX_TASKS_PER_NODE}
endif
@ taskpernode = ${MAX_TASKS_PER_NODE} / ${maxthrds}
#set qname = batch
set tlimit = "24:00:00"

#--- Job name is first fifteen characters of case name ---
set jobname = `echo ${CASE} | cut -c1-15`

if ($?TESTMODE) then
set file = $CASEROOT/${CASE}.test
else
set file = $CASEROOT/${CASE}.run
endif

cat >! $file << EOF1
#!/bin/csh -f
#===============================================================================
# USERDEFINED
# This is where the batch submission is set. The above code computes
# the total number of tasks, nodes, and other things that can be useful
# here. Use PBS, BSUB, or whatever the local environment supports.
#===============================================================================
#SBATCH --job-name ${jobname}
#SBATCH -N ${nodes}
#SBATCH --account=FY140052
#SBATCH --time=${tlimit}
#SBATCH --partition=ec


#limit coredumpsize 1000000
#limit stacksize unlimited


EOF1

#################################################################################
else if ($PHASE == set_exe) then
#################################################################################

source ./Tools/ccsm_getenv || exit -1
set maxthrds = `${CASEROOT}/Tools/taskmaker.pl -maxthrds`
set maxtasks = `${CASEROOT}/Tools/taskmaker.pl -sumtasks`


set ntasks = `${CASEROOT}/Tools/taskmaker.pl -sumonly`
@ nodes = $ntasks / ${MAX_TASKS_PER_NODE}
if ( $ntasks % ${MAX_TASKS_PER_NODE} > 0) then
@ nodes = $nodes + 1
@ ntasks = $nodes * ${MAX_TASKS_PER_NODE}
endif
@ taskpernode = ${MAX_TASKS_PER_NODE} / ${maxthrds}

cat >> ${CASEROOT}/${CASE}.run << EOF1
#sleep 25
cd \$RUNDIR
echo "\`date\` -- CSM EXECUTION BEGINS HERE"

setenv OMP_NUM_THREADS ${maxthrds}

#===============================================================================
# USERDEFINED
# edit job launching
#===============================================================================

#mpiexec -n ${maxtasks} \$EXEROOT/cesm.exe >&! ccsm.log.\$LID
#mpirun -np ${maxtasks} \$EXEROOT/cesm.exe >&! ccsm.log.\$LID
mpiexec -np ${maxtasks} \$EXEROOT/cesm.exe >&! cesm.log.\$LID

wait
echo "\`date\` -- CSM EXECUTION HAS FINISHED"

EOF1


#################################################################################
else if ($PHASE == set_larch) then
#################################################################################

#This is a place holder for a long-term archiving script


#################################################################################
else
#################################################################################

echo " PHASE setting of $PHASE is not an accepted value"
echo " accepted values are set_batch, set_exe and set_larch"
exit 1

#################################################################################
endif
#################################################################################
Loading