Excessive wall clock time for hera.gnu

[DeniseWorthen]

Description

Run-times for the current develop on hera.intel/gnu vs cheyenne.intel/gnu taken from the most Regression test logs.

control_p8:
hera.intel 165s
hera.gnu 890s

cheyenne.intel 178s
cheyenne.gnu 300s

cpld_control_p8:

hera.intel 333s
hera.gnu 1720s

cheyenne.intel 321s
cheyenne.gnu 495s

Cheyenne intel/gnu times are similar for both tests (but gnu is slower).

On hera, both control_p8 and cpld_control_p8 take ~5x longer for GNU.

[FernandoAndrade-NOAA]

@natalie-perlin Are there any current issues with utilizing gnu 10.2? It was suggested that would be a solution to the slower hera/gnu times

[jkbk2004]

@natalie-perlin Are there any current issues with utilizing gnu 10.2? It was suggested that would be a solution to the slower hera/gnu times

@natalie-perlin can we make sure /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-10.2/modulefiles/stack still up to date ? https://github.com/ufs-community/ufs-weather-model/blob/develop/modulefiles/ufs_common.lua

[natalie-perlin]

@jkbk2004 @FernandoAndrade-NOAA -
all verified, the /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-10.2/modulefiles/stack is up-to-date.

The following compiler flag needs to be added for gnu>=10 versions:

FFLAGS+="-fallow-argument-mismatch -fallow-invalid-boz"

[zach1221]

I'm not certain when this was last tested, but I tried again with the current spack-stack 1.4.1 installation on Hera and gnu seemed faster.
control_p8:
hera.intel 175s
hera.gnu 302s

cpld_control_p8:
hera.intel 351s
hera.gnu 651s

[DeniseWorthen]

@zach1221 We have >9 for GNU now on Hera w/ spack-stack, right? I think that was the issue previously (the GNU version).

If it now runs w/in the standard wall clock, we should remove this from the relevant tests

if [[ $MACHINE_ID = hera ]] && [[ $RT_COMPILER = gnu ]]; then
  WLCLK=40
fi

[zach1221]

@zach1221 We have >9 for GNU now on Hera w/ spack-stack, right? I think that was the issue previously (the GNU version).

If it now runs w/in the standard wall clock, we should remove this from the relevant tests

if [[ $MACHINE_ID = hera ]] && [[ $RT_COMPILER = gnu ]]; then
  WLCLK=40
fi

Ok, let me try removing that from the cpld_control_p8 and control_p8 tests now then retesting. If the result is the same maybe we can remove it in the next ufs-wm PR.

[zach1221]

Ok, I can confirm that cpld_control_p8 and control_p8 gnu continue to run between 600 - 650s for cpld_control_p8_gnu and between 300 - 260s for control_p8_gnu. This is with the standard wall clock. I have removed the hera.gnu wallclock setting from both test files in my ufs-wm #1912 . logs: /scratch1/NCEPDEV/nems/Zachary.Shrader/gnu_test

[zach1221]

PR 1946 has been merged. Closing this issue.