Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjustment to wave workflow for running wave component on Hera #82

Merged
merged 7 commits into from
Jun 4, 2020
Merged

Adjustment to wave workflow for running wave component on Hera #82

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
20ef779
Redefining mpmd command for working on Hera with slurm Adapting wave …
ajhenrique May 22, 2020
49abb90
Adding comment indicating how to regenerate cdo interpolation weights.
ajhenrique May 27, 2020
32c5f29
Merging latest feature/gfsv16b branch into feature/gfsv16b_wavehera
ajhenrique May 27, 2020
3393cac
Correcting minor bug in wave_prn_cur.sh Adjusting indents in exwave_p…
ajhenrique May 28, 2020
bb79d7a
Fixing a few minor bugs in wave_tar.sh after testing on WCOSS
ajhenrique Jun 3, 2020
ba89548
Update JWAVE_PREP
ajhenrique Jun 3, 2020
987c32a
Updating checkout to ufs-weather-model GFS.v16.0.5
ajhenrique Jun 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ export NTHSTACK=1024000000
export job=${PBS_JOBNAME:-$step}
export jobid=${job}.${PBS_JOBID:-$$}


if [ $step = "prep" -o $step = "prepbufr" ]; then

nth_max=$(($npe_node_max / $npe_node_prep))
Expand All @@ -37,6 +38,11 @@ if [ $step = "prep" -o $step = "prepbufr" ]; then
export BACK="NO"
export sys_tp="HERA"

elif [ $step = "waveinit" -o $step = "waveprep" -o $step = "wavepostsbs" ]; then

export mpmd="--multi-prog"
export CFP_MP="YES"

elif [ $step = "anal" ]; then

export MKL_NUM_THREADS=4
Expand Down
4 changes: 3 additions & 1 deletion jobs/JWAVE_PREP
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,9 @@ else
if [ ! -L $ROTDIR/${WAVECUR_DID}.${RPDY} ]; then # Check if symlink already exists in ROTDIR
$NLN $DMPDIR/${WAVECUR_DID}.${RPDY} $ROTDIR/${WAVECUR_DID}.${RPDY}
fi
$NLN $DMPDIR/$CDUMP.${PDY}/$cyc/${WAVICEFILE} $ROTDIR/$CDUMP.${PDY}/$cyc/${WAVICEFILE}
if [ ! -L $ROTDIR/${CDUMP}.${PDY}/${cyc}/${WAVICEFILE} ]; then # Check if symlink already exists in ROTDIR
ajhenrique marked this conversation as resolved.
Show resolved Hide resolved
$NLN $DMPDIR/$CDUMP.${PDY}/$cyc/${WAVICEFILE} $ROTDIR/$CDUMP.${PDY}/$cyc/${WAVICEFILE}
fi
export COMIN_OBS=${COMIN_OBS:-$ROTDIR/$RUN.$PDY/$cyc}
export COMIN_WAV_ICE=${COMIN_OBS}
export COMIN_WAV_WND=${COMIN_OBS}
Expand Down
3 changes: 3 additions & 0 deletions modulefiles/module_base.hera
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,6 @@ module load prod_util/1.1.0
# python
module use -a /contrib/modulefiles
module load anaconda/2.3.0

# waveprep
module load cdo/1.9.5
4 changes: 2 additions & 2 deletions parm/config/config.waveprep
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@ export WAV_WND_HOUR_INC=1 # This value should match with the one used in
# the wind update script
# Intake currents settings
export WAV_CUR_DT=${WAV_CUR_DT:-3}
export WAV_CUR_HF_DT=${WAV_CUR_HF_DT:-3}
export WAV_CUR_HF_FH=${WAV_CUR_HF_FH:-72}
export WAV_CUR_HF_DT=${WAV_CUR_HF_DT:-3} # Defaults to 3h for GEFSv12
export WAV_CUR_HF_FH=${WAV_CUR_HF_FH:-0} # Constant DT for GFSv16 from getgo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean WAVE currently running in the parallels has a wrong setting?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please confirm ?

export WAV_CUR_CDO_SMOOTH="NO"

# Location of CDO module
Expand Down
12 changes: 10 additions & 2 deletions scripts/exwave_init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,11 @@
fi

[[ ! -d $COMOUT/rundata ]] && mkdir -m 775 -p $COMOUT/rundata
echo "$USHwave/wave_grid_moddef.sh $grdID > $grdID.out 2>&1" >> cmdfile
if [ ${CFP_MP:-"NO"} = "YES" ]; then
echo "$nmoddef $USHwave/wave_grid_moddef.sh $grdID > $grdID.out 2>&1" >> cmdfile
ajhenrique marked this conversation as resolved.
Show resolved Hide resolved
else
echo "$USHwave/wave_grid_moddef.sh $grdID > $grdID.out 2>&1" >> cmdfile
fi

nmoddef=`expr $nmoddef + 1`

Expand Down Expand Up @@ -162,7 +166,11 @@

if [ "$NTASKS" -gt '1' ]
then
${wavempexec} ${wavenproc} ${wave_mpmd} cmdfile
if [ ${CFP_MP:-"NO"} = "YES" ]; then
${wavempexec} -n ${wavenproc} ${wave_mpmd} cmdfile
else
${wavempexec} ${wavenproc} ${wave_mpmd} cmdfile
fi
exit=$?
else
./cmdfile
Expand Down
85 changes: 42 additions & 43 deletions scripts/exwave_prep.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -638,55 +638,55 @@
touch cmdfile
chmod 744 cmdfile

ymdh_rtofs=${PDY}00 # RTOFS runs once daily
ymdh_end=`$NDATE ${FHMAX_WAV_CUR} ${ymdh_rtofs}`
NDATE_DT=${WAV_CUR_HF_DT}
FLGHF='T'
ymdh_rtofs=${PDY}00 # RTOFS runs once daily use ${PDY}00
ymdh_end=`$NDATE ${FHMAX_WAV_CUR} ${PDY}00`
NDATE_DT=${WAV_CUR_HF_DT}
FLGHF='T'

if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=0 ; fi # Counter for MP CFP
while [ "$ymdh_rtofs" -le "$ymdh_end" ]
do
if [ ${CFP_MP:-"NO"} = "YES" ]; then nm=0 ; fi # Counter for MP CFP
while [ "$ymdh_rtofs" -le "$ymdh_end" ]
do
# Timing has to be made relative to the single 00z RTOFS cycle for that PDY
fhr_rtofs=`${NHOUR} ${ymdh_rtofs} ${PDY}00`
fext='f'
fhr_rtofs=`${NHOUR} ${ymdh_rtofs} ${PDY}00`
fext='f'

fh3_rtofs=`printf "%03d" "${fhr_rtofs#0}"`
fh3_rtofs=`printf "%03d" "${fhr_rtofs#0}"`

curfile1h=${COMIN_WAV_CUR}/rtofs_glo_2ds_${fext}${fh3_rtofs}_1hrly_prog.nc
curfile3h=${COMIN_WAV_CUR}/rtofs_glo_2ds_${fext}${fh3_rtofs}_3hrly_prog.nc
curfile1h=${COMIN_WAV_CUR}/rtofs_glo_2ds_${fext}${fh3_rtofs}_1hrly_prog.nc
curfile3h=${COMIN_WAV_CUR}/rtofs_glo_2ds_${fext}${fh3_rtofs}_3hrly_prog.nc

if [ -s ${curfile1h} ] && [ "${FLGHF}" = "T" ] ; then
curfile=${curfile1h}
elif [ -s ${curfile3h} ]; then
curfile=${curfile3h}
FLGHF='F'
else
echo ' '
set $setoff
echo ' '
echo '************************************** '
echo "*** FATAL ERROR: NO CUR FILE $curfile *** "
echo '************************************** '
echo ' '
set $seton
postmsg "$jlogfile" "FATAL ERROR - NO CURRENT FILE (RTOFS)"
err=11;export err;${errchk}
exit 0
echo ' '
fi
if [ -s ${curfile1h} ] && [ "${FLGHF}" = "T" ] ; then
curfile=${curfile1h}
elif [ -s ${curfile3h} ]; then
curfile=${curfile3h}
FLGHF='F'
else
echo ' '
set $setoff
echo ' '
echo '************************************** '
echo "*** FATAL ERROR: NO CUR FILE $curfile *** "
echo '************************************** '
echo ' '
set $seton
postmsg "$jlogfile" "FATAL ERROR - NO CURRENT FILE (RTOFS)"
err=11;export err;${errchk}
exit 0
echo ' '
fi

if [ ${CFP_MP:-"NO"} = "YES" ]; then
echo "$nm $USHwave/wave_prnc_cur.sh $ymdh_rtofs $curfile $fhr_rtofs > cur_$ymdh_rtofs.out 2>&1" >> cmdfile
nm=`expr $nm + 1`
else
echo "$USHwave/wave_prnc_cur.sh $ymdh_rtofs $curfile $fhr_rtofs > cur_$ymdh_rtofs.out 2>&1" >> cmdfile
fi
if [ ${CFP_MP:-"NO"} = "YES" ]; then
echo "$nm $USHwave/wave_prnc_cur.sh $ymdh_rtofs $curfile $fhr_rtofs > cur_$ymdh_rtofs.out 2>&1" >> cmdfile
nm=`expr $nm + 1`
else
echo "$USHwave/wave_prnc_cur.sh $ymdh_rtofs $curfile $fhr_rtofs > cur_$ymdh_rtofs.out 2>&1" >> cmdfile
fi

if [ $fhr_rtofs -ge ${WAV_CUR_HF_FH} ] ; then
NDATE_DT=${WAV_CUR_DT}
fi
ymdh_rtofs=`$NDATE $NDATE_DT $ymdh_rtofs`
done
if [ $fhr_rtofs -ge ${WAV_CUR_HF_FH} ] ; then
NDATE_DT=${WAV_CUR_DT}
fi
ymdh_rtofs=`$NDATE $NDATE_DT $ymdh_rtofs`
done

# Set number of processes for mpmd
wavenproc=`wc -l cmdfile | awk '{print $1}'`
Expand Down Expand Up @@ -747,7 +747,6 @@
do
echo $file
cat $file >> cur.${WAVECUR_FID}
rm -f $file
done

cp -f cur.${WAVECUR_FID} ${COMOUT}/rundata/${COMPONENTwave}.${WAVECUR_FID}.$cycle.cur
Expand Down
2 changes: 1 addition & 1 deletion sorc/checkout.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ if [[ ! -d fv3gfs.fd ]] ; then
rm -f ${topdir}/checkout-fv3gfs.log
git clone https://github.com/ufs-community/ufs-weather-model fv3gfs.fd >> ${topdir}/checkout-fv3gfs.log 2>&1
cd fv3gfs.fd
git checkout GFS.v16.0.4
git checkout GFS.v16.0.5
git submodule update --init --recursive
cd ${topdir}
else
Expand Down
2 changes: 1 addition & 1 deletion ush/wave_prnc_cur.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ fi
# Cleanup
rm -f cur_temp[123].nc cur_5min_??.nc cur_glo_uv_${PDY}_${fext}${fh3}.nc weights.nc

if [ ${fhr} -gt ${WAVHINDH} ]
if [ ${fhr} -gt 0 ]
then
sed -e "s/HDRFL/F/g" ${FIXwave}/ww3_prnc.cur.${WAVECUR_FID}.inp.tmpl > ww3_prnc.inp
else
Expand Down
9 changes: 4 additions & 5 deletions ush/wave_tar.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@

# Use LOUD variable to turn on/off trace. Defaults to YES (on).
export LOUD=${LOUD:-YES}; [[ $LOUD = yes ]] && export LOUD=YES
[[ "$LOUD" != YES ]] && set +x
[[ "$LOUD" != YES ]] && set -x
yangfanglin marked this conversation as resolved.
Show resolved Hide resolved

cd $DATA
postmsg "$jlogfile" "Making TAR FILE"
Expand Down Expand Up @@ -98,6 +98,7 @@
set +x
echo ' '
echo ' Making tar file ...'
set -x

count=0
countMAX=5
Expand All @@ -106,14 +107,12 @@
while [ "$count" -lt "$countMAX" ] && [ "$tardone" = 'no' ]
do

[[ "$LOUD" = YES ]] && set -v
# JY nf=`ls $ID.*.$type | wc -l | awk '{ print $1 }'`
nf=`ls | awk '/'$ID.*.$filext'/ {a++} END {print a}'`
if [ "$nf" = "$nb" ]
nbm2=$(( $nb - 2 ))
if [ $nf -ge $nbm2 ]
then
tar -cf $ID.$cycle.${type}_tar ./$ID.*.$filext
exit=$?
set +v; [[ "$LOUD" = YES ]] && set -x

if [ "$exit" != '0' ]
then
Expand Down