Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Met plus tc #31

Closed
wants to merge 11 commits into from
Closed

Met plus tc #31

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
4a6daea
MetPlusTC for EVSv1.0.0
JiayiPeng-NOAA Nov 3, 2022
4c3f680
MetPlusTC for EVSv1.0.0
JiayiPeng-NOAA Nov 3, 2022
b8b2127
MetPlusTC for EVSv1.0.0
JiayiPeng-NOAA Nov 3, 2022
35237e0
Update COMINs in ecf/jobs files
JiayiPeng-NOAA Nov 8, 2022
277bcf6
Merge 3-jobs into one.
JiayiPeng-NOAA Nov 15, 2022
16c4647
Update the J-job.
JiayiPeng-NOAA Nov 16, 2022
54e2f4f
shapefile,new version and regional added
JiayiPeng-NOAA Dec 3, 2022
f8502c9
Update HOWtoRUN
JiayiPeng-NOAA Dec 3, 2022
2ed6278
para to prod
JiayiPeng-NOAA Dec 7, 2022
6c4c1bd
update HOWtoRUN
JiayiPeng-NOAA Dec 7, 2022
e54517f
Merge branch 'MetPlusTC' of https://github.com/JiayiPeng-NOAA/EVS int…
JiayiPeng-NOAA Dec 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions HOWtoRUN
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
cd ./ecf/hurricane/stats
(1) change your job Account for each ecf files
(2) qsub jevs_hurricane_global_det_tcgen_stats.ecf
qsub jevs_hurricane_global_det_tropcyc_stats.ecf
qsub jevs_hurricane_global_ens_tropcyc_stats.ecf
(3) See the stats file and plots under:
/lfs/h2/emc/ptmp/$USER/com/evs
73 changes: 73 additions & 0 deletions ecf/hurricane/stats/jevs_hurricane_global_det_tcgen_stats.ecf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#PBS -S /bin/bash
#PBS -N jevs_hurricane_global_det_tcgen_stats
#PBS -j oe
#PBS -A ENSTRACK-DEV
#PBS -q dev
#PBS -l select=1:ncpus=2:mem=4GB
##PBS -l place=vscatter:exclhost,select=1:ncpus=128:ompthreads=1
#PBS -l walltime=00:30:00
#PBS -l debug=true
#PBS -V

#%include <head.h>
#%include <envir-p1.h>

############################################################
# Load modules
############################################################
module purge
export HPC_OPT=/apps/ops/para/libs
module use /apps/ops/para/libs/modulefiles/compiler/intel/19.1.3.304/
export HOMEevs=/lfs/h2/emc/vpppg/save/$USER/EVS
source ${HOMEevs}/versions/run.ver
module load intel/${intel_ver}
module load gsl/${gsl_ver}
module load python/${python_ver}
module load netcdf/${netcdf_ver}
module load met/${met_ver}
module load metplus/${metplus_ver}

module load libjpeg/$libjpeg_ver
module load grib_util/$grib_util_ver
module load prod_util/${produtil_ver}
module load prod_envir/${prodenvir_ver}

module load geos/${geos_ver}
module load proj/${proj_ver}
module load libfabric/${libfabric}
module load imagemagick/${imagemagick_ver}

module list

export NET=evs
export COMPONENT=hurricane_global_det
export RUN=tcgen
export STEP=stats

export envir=dev
export cyc=00
export job=jevs_hurricane_global_det_tcgen_stats_${cyc}

#Define the directories of your TC genesis data and A/Bdeck files
export COMINgenesis=/lfs/h2/emc/vpppg/save/jiayi.peng/TCgen/genesisDATA
export COMINadeckNHC=/lfs/h2/emc/vpppg/save/jiayi.peng/TCgen/ABdeck
export COMINbdeckNHC=/lfs/h2/emc/vpppg/save/jiayi.peng/TCgen/ABdeck
export COMINadeckJTWC=/lfs/h2/emc/vpppg/save/jiayi.peng/TCgen/ABdeck
export COMINbdeckJTWC=/lfs/h2/emc/vpppg/save/jiayi.peng/TCgen/ABdeck

export DATAROOT=/lfs/h2/emc/ptmp/$USER
export COMROOT=${DATAROOT}/com
export KEEPDATA=YES


# CALL executable job script here
$HOMEevs/jobs/hurricane/stats/JEVS_HURRICANE_STATS

%include <tail.h>
%manual
######################################################################
# Purpose: This job will generate the grid2obs statistics for the HRRR
# model and generate stat files.
######################################################################
%end

68 changes: 68 additions & 0 deletions ecf/hurricane/stats/jevs_hurricane_global_det_tropcyc_stats.ecf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#PBS -S /bin/bash
#PBS -N jevs_hurricane_global_det_tropcyc_stats
#PBS -j oe
#PBS -A ENSTRACK-DEV
#PBS -q dev
#PBS -l select=1:ncpus=2:mem=4GB
##PBS -l place=vscatter:exclhost,select=1:ncpus=128:ompthreads=1
#PBS -l walltime=00:30:00
#PBS -l debug=true
#PBS -V

#%include <head.h>
#%include <envir-p1.h>

############################################################
# Load modules
############################################################
module purge
export HPC_OPT=/apps/ops/para/libs
module use /apps/ops/para/libs/modulefiles/compiler/intel/19.1.3.304/
export HOMEevs=/lfs/h2/emc/vpppg/save/$USER/EVS
source ${HOMEevs}/versions/run.ver
module load intel/${intel_ver}
module load gsl/${gsl_ver}
module load python/${python_ver}
module load netcdf/${netcdf_ver}
module load met/${met_ver}
module load metplus/${metplus_ver}

module load libjpeg/$libjpeg_ver
module load grib_util/$grib_util_ver
module load prod_util/${produtil_ver}
module load prod_envir/${prodenvir_ver}
module load imagemagick/${imagemagick_ver}

module list

export NET=evs
export COMPONENT=hurricane_global_det
export RUN=tropcyc
export STEP=stats

export envir=dev
export cyc=00
export job=jevs_hurricane_global_det_tropcyc_stats_${cyc}

#Define TC-vital file, TC track file and the directory for Bdeck files
export COMINvit=/lfs/h1/ops/prod/com/gfs/v16.2/syndat/syndat_tcvitals.2022
export COMINtrack=/lfs/h1/ops/prod/com/ens_tracker/v1.3/global/tracks.atcfunix.22
export COMINbdeckNHC=/lfs/h1/ops/prod/dcom/nhc/atcf-noaa/btk
export COMINbdeckJTWC=/lfs/h1/ops/prod/dcom/nhc/atcf-navy/btk

export DATAROOT=/lfs/h2/emc/ptmp/$USER
export COMROOT=${DATAROOT}/com
export KEEPDATA=YES


# CALL executable job script here
$HOMEevs/jobs/hurricane/stats/JEVS_HURRICANE_STATS

%include <tail.h>
%manual
######################################################################
# Purpose: This job will generate the grid2obs statistics for the HRRR
# model and generate stat files.
######################################################################
%end

68 changes: 68 additions & 0 deletions ecf/hurricane/stats/jevs_hurricane_global_ens_tropcyc_stats.ecf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#PBS -S /bin/bash
#PBS -N jevs_hurricane_global_ens_tropcyc_stats
#PBS -j oe
#PBS -A ENSTRACK-DEV
#PBS -q dev
#PBS -l select=1:ncpus=2:mem=4GB
##PBS -l place=vscatter:exclhost,select=1:ncpus=128:ompthreads=1
#PBS -l walltime=00:30:00
#PBS -l debug=true
#PBS -V

#%include <head.h>
#%include <envir-p1.h>

############################################################
# Load modules
############################################################
module purge
export HPC_OPT=/apps/ops/para/libs
module use /apps/ops/para/libs/modulefiles/compiler/intel/19.1.3.304/
export HOMEevs=/lfs/h2/emc/vpppg/save/$USER/EVS
source ${HOMEevs}/versions/run.ver
module load intel/${intel_ver}
module load gsl/${gsl_ver}
module load python/${python_ver}
module load netcdf/${netcdf_ver}
module load met/${met_ver}
module load metplus/${metplus_ver}

module load libjpeg/$libjpeg_ver
module load grib_util/$grib_util_ver
module load prod_util/${produtil_ver}
module load prod_envir/${prodenvir_ver}
module load imagemagick/${imagemagick_ver}

module list

export NET=evs
export COMPONENT=hurricane_global_ens
export RUN=tropcyc
export STEP=stats

export envir=dev
export cyc=00
export job=jevs_hurricane_global_ens_tropcyc_stats_${cyc}

#Define TC-vital file, TC track file and the directory for Bdeck files
export COMINvit=/lfs/h1/ops/prod/com/gfs/v16.2/syndat/syndat_tcvitals.2022
export COMINtrack=/lfs/h1/ops/prod/com/ens_tracker/v1.3/global/tracks.atcfunix.22
export COMINbdeckNHC=/lfs/h1/ops/prod/dcom/nhc/atcf-noaa/btk
export COMINbdeckJTWC=/lfs/h1/ops/prod/dcom/nhc/atcf-navy/btk

export DATAROOT=/lfs/h2/emc/ptmp/$USER
export COMROOT=${DATAROOT}/com
export KEEPDATA=YES


# CALL executable job script here
$HOMEevs/jobs/hurricane/stats/JEVS_HURRICANE_STATS

%include <tail.h>
%manual
######################################################################
# Purpose: This job will generate the grid2obs statistics for the HRRR
# model and generate stat files.
######################################################################
%end

Binary file added fix/noaa.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added fix/nws.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
135 changes: 134 additions & 1 deletion jobs/hurricane/stats/JEVS_HURRICANE_STATS
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,134 @@
sample
#!/bin/bash
date
export PS4=' $SECONDS + '
set -x

####################################
# obtain unique process id (pid) and make temp directory
####################################
export jobid=${jobid:-$job.o$$}
export DATA=${DATA:-${DATAROOT:?}/${jobid}}
mkdir -p $DATA
cd $DATA
export cycle=t${cyc}z

####################################
# File To Log Msgs
####################################
export jlogfile=${jlogfile:-${DATA}/jlogfile.${jobid}}

####################################
# Specify NET and RUN Name and model
####################################
export NET=${NET:-evs}
export COMPONENT=${COMPONENT:-define_in_ecf}
export RUN=${RUN:-define_in_ecf}
Copy link
Contributor

@ShelleyMelchior-NOAA ShelleyMelchior-NOAA Nov 16, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the default of 'define_in_ecf' will cause this J-job to break b/c a test for 'define_in_ecf' is not in place in the if-elif-fi below. I suggest that you hard code a default for COMPONENT and RUN that will not cause the J-job to break. For example, 'hurricane_global_det' for COMPONENT and 'tropcyc' for RUN.
Alternatively, you can run a quick test right here to make sure that COMPONENT and RUN are previously set to acceptable values, and if not, then force a graceful exit. A check like this is important if certain variables are required to be defined.

export STEP=${STEP:-stats}

####################################
# Determine Job Output Name on System
####################################
export pgmout="OUTPUT.$$"
export pgmerr=errfile

####################################
# SENDECF - Flag Events on ecFLOW
# SENDCOM - Copy Files From TMPDIR to $COMOUT
# SENDDBN - Issue DBNet Client Calls
####################################
export SENDECF=${SENDECF:-NO}
export SENDCOM=${SENDCOM:-YES}
export SENDDBN=${SENDDBN:-NO}

####################################
# Specify Execution Areas
####################################
export HOMEevs=${HOMEevs:-${PACKAGEHOME}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is $PACKAGEHOME defined? This will break if $HOMEevs is not defined in the ecf script.

export SCRIPTSevs=${SCRIPTSevs:-$HOMEevs/scripts/hurricane/stats}
export USHevs=${USHevs:-$HOMEevs/ush/hurricane/stats}
export PARMevs=${PARMevs:-$HOMEevs/parm/metplus_config/hurricane/stats}
export FIXevs=${FIXevs:-$HOMEevs/fix}

# Run setpdy and initialize PDY variables
##############################
#setpdy.sh
#. ./PDY

##############################################
# Define COM directories
##############################################
if [ ${COMPONENT} = "hurricane_global_det" -a ${RUN} = "tropcyc" ]; then
setpdy.sh
. ./PDY
export YYYY=`echo ${PDY} | cut -c1-4`
export YY22=`echo ${PDY} | cut -c3-4`

export COMINvit=${COMINvit:-/your/TC/vitals/file}
export COMINtrack=${COMINtrack:-your/TC/track/file}
export COMINbdeckNHC=${COMINbdeckNHC:-/your/NHC/bdeck/data/dir}
export COMINbdeckJTWC=${COMINbdeckJTWC:-/your/JTWC/bdeck/data/dir}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This kind of default path, e.g., '/your/TC/vintals/file', or '/your/TC/track/file', or '/your/NHC/bdeck/data/dir', or '/your/JTWC/bdeck/data/dir', will cause the downstream processing to have problems if these COMIN* variables are not defined in the ecf script.
We either need to use a default path that indeed exists on the computing system that the verification package is running on, or we need to require that these COMIN* variables be defined in order to proceed. If not defined, gracefully exit.


elif [ ${COMPONENT} = "hurricane_global_ens" -a ${RUN} = "tropcyc" ]; then
setpdy.sh
. ./PDY
export YYYY=`echo ${PDY} | cut -c1-4`
export YY22=`echo ${PDY} | cut -c3-4`

export COMINvit=${COMINvit:-/your/TC/vitals/file}
export COMINtrack=${COMINtrack:-your/TC/track/file}
export COMINbdeckNHC=${COMINbdeckNHC:-/your/NHC/bdeck/data/dir}
export COMINbdeckJTWC=${COMINbdeckJTWC:-/your/JTWC/bdeck/data/dir}

elif [ ${COMPONENT} = "hurricane_global_det" -a ${RUN} = "tcgen" ]; then
#setpdy.sh
#. ./PDY
export PDY=20211115

export YYYY=`echo ${PDY} | cut -c1-4`
#export YY22=`echo ${PDY} | cut -c3-4`
export COMINgenesis=${COMINgenesis:-/your/TC/genesis/data/dir}
export COMINadeckNHC=${COMINadeckNHC:-/your/NHC/adeck/data/dir}
export COMINbdeckNHC=${COMINbdeckNHC:-/your/NHC/bdeck/data/dir}
export COMINadeckJTWC=${COMINadeckJTWC:-/your/JTWC/adeck/data/dir}
export COMINbdeckJTWC=${COMINbdeckJTWC:-/your/JTWC/bdeck/data/dir}

elif [ ${COMPONENT} = "hurricane_regional" -a ${RUN} = "tropcyc" ]; then
setpdy.sh
. ./PDY
export YYYY=`echo ${PDY} | cut -c1-4`
export YY22=`echo ${PDY} | cut -c3-4`

export COMINvit=${COMINvit:-/your/TC/vitals/file}
export COMINtrack=${COMINtrack:-your/TC/track/file}
export COMINbdeckNHC=${COMINbdeckNHC:-/your/NHC/bdeck/data/dir}
export COMINbdeckJTWC=${COMINbdeckJTWC:-/your/JTWC/bdeck/data/dir}

fi

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire if-elif-fi block is good! But I think we need one final catch to allow the job to exit gracefully should none of the cases be satisfied. This is to catch the off chance for a user typo, for example, in the ecf script when defining $COMPONENT and $RUN.

#export COMOUT=${COMOUT:-$(compath.py -o ${NET}/${evs_ver})/${COMPONENT}/${RUN}/${STEP}}
export COMOUT=${COMOUT:-${COMROOT}/${NET}/${evs_ver}/${COMPONENT}/${RUN}/${STEP}}
mkdir -m 775 -p $COMOUT

msg="HAS BEGUN on `hostname`"
postmsg "$jlogfile" "$msg"

#env

#############################################################
# Execute the script
${SCRIPTSevs}/exevs_${COMPONENT}_${RUN}_stats.sh
export err=$?; err_chk

#############################################################

msg="JOB COMPLETED NORMALLY"
postmsg "$jlogfile" "$msg"

##############################
# Remove the Temporary working directory
##############################
if [[ $KEEPDATA != "YES" ]]; then
rm -rf $DATA
fi

date
Loading