Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UFS-dev PR#27 #81

Merged
merged 16 commits into from
Jan 3, 2023
Merged

UFS-dev PR#27 #81

merged 16 commits into from
Jan 3, 2023

Conversation

grantfirl
Copy link
Collaborator

Identical to ufs-community#1518

JianpingHuang-NOAA and others added 5 commits December 7, 2022 09:13
* adding hr_ave_chem_v1

* aqm diag_table changes

Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
Co-authored-by: JONG KIM <jong.kim@noaa.gov>
* removed control_2dwrtdecomp, control_fhzero, control_2threads_debug
…unity#1514)

* Update Jenkinsfile

* Update ci.test

* add a mixed-mode test

Co-authored-by: zach1221 <99902696+zach1221@users.noreply.github.com>
Co-authored-by: JONG KIM <jong.kim@noaa.gov>
Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
Co-authored-by: jkbk2004 <jkbk2004@gmail.com>
…munity#1518)

* Performance optimization of moving nest.

* Turn on HAFS RTs on WCOSS2 and switch to use the netcdf (instead of
netcdf_parallel) write_grid_component option for HAFS RTs.

* Update submodule FV3, which points to latest UPP develop branch.

* Update submodule FV3, which will only call atmosphere_fill_nest_cpl at the cap
driver time steps (coupling time steps)

* Adding upoff as a namelist parameter

* Update submodule HYCOM after the momentum flux fix for HAFS FV3ATM-HYCOM
coupling.

* Update submodule FV3 and rename HAFS_V0 CCPP physics suites into HAFS_V1.

* Cleanup of CMakefile for moving nest, and added fv_timers namelist option

Co-authored-by: William Ramstrom <William.Ramstrom@noaa.gov>
Co-authored-by: Biju Thomas <biju.thomas@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
Co-authored-by: jkbk2004 <jong.kim@noaa.gov>
@grantfirl
Copy link
Collaborator Author

Contains changes from #79 and #80 until those are merged. Also ufs-community#1505, ufs-community#1513, ufs-community#1525, ufs-community#1514 (all not related to CCPP)

@grantfirl
Copy link
Collaborator Author

New baselines needed for HAFS-related RTs.

@grantfirl
Copy link
Collaborator Author

Baselines also changed in ufs-community#1513

@grantfirl
Copy link
Collaborator Author

ufs-community#1514 adds a new RT too

dustinswales added 2 commits December 26, 2022 09:02
on-behalf-of @NCAR <dswales@ucar.edu>
on-behalf-of @NCAR <dswales@ucar.edu>
@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: RT
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20221226084509/ufs-weather-model
[RT] Error: Test cpld_control_p8_mixedmode 001 failed in check_result failed
[RT] Error: Test cpld_control_p8_mixedmode 001 failed in run_test failed
[RT] Error: Test hafs_regional_atm_ocn 114 failed in check_result failed
[RT] Error: Test hafs_regional_atm_ocn 114 failed in run_test failed
[RT] Error: Test hafs_regional_atm_ocn_wav 116 failed in check_result failed
[RT] Error: Test hafs_regional_atm_ocn_wav 116 failed in run_test failed
[RT] Error: Test hafs_regional_1nest_atm 117 failed in check_result failed
[RT] Error: Test hafs_regional_1nest_atm 117 failed in run_test failed
[RT] Error: Test hafs_regional_telescopic_2nests_atm 118 failed in check_result failed
[RT] Error: Test hafs_regional_telescopic_2nests_atm 118 failed in run_test failed
[RT] Error: Test hafs_global_1nest_atm 119 failed in check_result failed
[RT] Error: Test hafs_global_1nest_atm 119 failed in run_test failed
[RT] Error: Test hafs_global_multiple_4nests_atm 120 failed in check_result failed
[RT] Error: Test hafs_global_multiple_4nests_atm 120 failed in run_test failed
[RT] Error: Test hafs_regional_specified_moving_1nest_atm 121 failed in check_result failed
[RT] Error: Test hafs_regional_specified_moving_1nest_atm 121 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm 122 failed in check_result failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm 122 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn 123 failed in check_result failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn 123 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 124 failed in check_result failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 124 failed in run_test failed
[RT] Error: Test hafs_global_storm_following_1nest_atm 125 failed in check_result failed
[RT] Error: Test hafs_global_storm_following_1nest_atm 125 failed in run_test failed
[RT] Error: Test hafs_regional_datm_cdeps 128 failed in check_result failed
[RT] Error: Test hafs_regional_datm_cdeps 128 failed in run_test failed
[RT] Error: Test regional_atmaq 150 failed in check_result failed
[RT] Error: Test regional_atmaq 150 failed in run_test failed
[RT] Error: Test regional_atmaq_debug 151 failed in check_result failed
[RT] Error: Test regional_atmaq_debug 151 failed in run_test failed
Please make changes and add the following label back: cheyenne-intel-RT

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: intel
Job: RT
[RT] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1172606845/20221226154513/ufs-weather-model
[RT] Error: Test cpld_control_p8_mixedmode 001 failed in check_result failed
[RT] Error: Test cpld_control_p8_mixedmode 001 failed in run_test failed
[RT] Error: Test hafs_regional_atm_ocn 116 failed in check_result failed
[RT] Error: Test hafs_regional_atm_ocn 116 failed in run_test failed
[RT] Error: Test hafs_regional_atm_ocn_wav 118 failed in check_result failed
[RT] Error: Test hafs_regional_atm_ocn_wav 118 failed in run_test failed
[RT] Error: Test hafs_regional_1nest_atm 119 failed in check_result failed
[RT] Error: Test hafs_regional_1nest_atm 119 failed in run_test failed
[RT] Error: Test hafs_regional_telescopic_2nests_atm 120 failed in check_result failed
[RT] Error: Test hafs_regional_telescopic_2nests_atm 120 failed in run_test failed
[RT] Error: Test hafs_global_1nest_atm 121 failed in check_result failed
[RT] Error: Test hafs_global_1nest_atm 121 failed in run_test failed
[RT] Error: Test hafs_global_multiple_4nests_atm 122 failed in check_result failed
[RT] Error: Test hafs_global_multiple_4nests_atm 122 failed in run_test failed
[RT] Error: Test hafs_regional_specified_moving_1nest_atm 123 failed in check_result failed
[RT] Error: Test hafs_regional_specified_moving_1nest_atm 123 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm 124 failed in check_result failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm 124 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn 125 failed in check_result failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn 125 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 126 failed in check_result failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 126 failed in run_test failed
[RT] Error: Test hafs_global_storm_following_1nest_atm 127 failed in check_result failed
[RT] Error: Test hafs_global_storm_following_1nest_atm 127 failed in run_test failed
[RT] Error: Test hafs_regional_datm_cdeps 130 failed in check_result failed
[RT] Error: Test hafs_regional_datm_cdeps 130 failed in run_test failed
[RT] Error: Test regional_atmaq 152 failed in check_result failed
[RT] Error: Test regional_atmaq 152 failed in run_test failed
[RT] Error: Test regional_atmaq_debug 153 failed in check_result failed
[RT] Error: Test regional_atmaq_debug 153 failed in run_test failed
Please make changes and add the following label back: hera-intel-RT

@grantfirl
Copy link
Collaborator Author

@dustinswales It looks like the expected RTs fail (HAFS, mixed-mode, and atmaq tests). I'll change BL_DATE in rt.ncar.sh and add the BL labels.

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: cheyenne
Compiler: gnu
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20221230114507/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20221230121447/ufs-weather-model
Please make changes and add the following label back: cheyenne-gnu-BL

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20221230120008/ufs-weather-model
[BL] Error: Test regional_atmaq_debug 116 failed in run_test failed
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: gnu
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1172606845/20221230184515/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1172606845/20221230193054/ufs-weather-model
[RT] Error: Test cpld_control_p8 046 failed in run_test failed
Please make changes and add the following label back: hera-gnu-BL

on-behalf-of @NCAR <dswales@ucar.edu>
@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: intel
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1172606845/20221230190011/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1172606845/20221230200146/ufs-weather-model
Please make changes and add the following label back: hera-intel-BL

@grantfirl
Copy link
Collaborator Author

FYI, @dustinswales I'm trying something new. To get around the failed BL creation, I changed the BL_DATE in rt.ncar.sh to 20221230a rather than 20221230 so that it will create a different folder and not error about the dir already existing. This is a workaround for me not being able to delete files in your space manually. If this works, you'll just need to manually copy the new BLs from the 20221230a directory to the 20221230 directory and I'll revert the BL_DATE back to 20221230 when I revert the WLCLK changes.

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20221230144507/ufs-weather-model
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Collaborator

FYI, @dustinswales I'm trying something new. To get around the failed BL creation, I changed the BL_DATE in rt.ncar.sh to 20221230a rather than 20221230 so that it will create a different folder and not error about the dir already existing. This is a workaround for me not being able to delete files in your space manually. If this works, you'll just need to manually copy the new BLs from the 20221230a directory to the 20221230 directory and I'll revert the BL_DATE back to 20221230 when I revert the WLCLK changes.

@grantfirl Sorry, I thought I changed the permissions for you to be able to write (I'll check on this) But this seems like a good idea.

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: hera
Compiler: gnu
Job: BL
[BL] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1172606845/20221230214507/ufs-weather-model
Please make changes and add the following label back: hera-gnu-BL

@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20230102141507/ufs-weather-model
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Collaborator

@grantfirl Looks like that "a" suffix causes problems:
INFO:BL/UPDATE_RT_NCAR_SH:Date 20221230a is not formatted YYYYMMDD
CRITICAL:JOB/RUN:run() FAILED. Exception:<class 'Exception'>

We're almost there... All the tests pass except Cheyenne Intel. I cleaned up that directory (/glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/NCAR/main-20221230/INTEL). Whenever you get a chance revert the BL_DATE change and kick off a new cheyenne-intel-BL.

on-behalf-of @NCAR <dswales@ucar.edu>
@dustinswales
Copy link
Collaborator

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20230103103011/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1172606845/20230103113701/ufs-weather-model
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Collaborator

@grantfirl It looks like the test worked, but also failed?
Looking at the log, /glade/u/home/dswales/Projects/UFS/auto-RT/ufs-weather-model/tests/auto/rt_auto_20230103103009.log, it runs the RTs and pushes the RT logs, but then fails? Looking at the script, rt_auto.py (same dir as above), it seems that it is failing when trying to remove the github label?

@grantfirl grantfirl requested a review from dustinswales January 3, 2023 20:40
@grantfirl
Copy link
Collaborator Author

@grantfirl It looks like the test worked, but also failed? Looking at the log, /glade/u/home/dswales/Projects/UFS/auto-RT/ufs-weather-model/tests/auto/rt_auto_20230103103009.log, it runs the RTs and pushes the RT logs, but then fails? Looking at the script, rt_auto.py (same dir as above), it seems that it is failing when trying to remove the github label?

Ya, it's been spitting out error messages to GitHub even after successful RTs. I don't know how it could detect failure on removing the label, because that was successful too. I think that we should be good to go ahead and merge this and try to figure out the failure messages afterward? If you agree, please approve this and the fv3atm PR so that we can merge.

@grantfirl grantfirl merged commit 4b1c00c into NCAR:main Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants