Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add in ufsda python tools and example j-jobs/ex-scripts #19

Merged
merged 41 commits into from
Mar 24, 2022
Merged

Add in ufsda python tools and example j-jobs/ex-scripts #19

merged 41 commits into from
Mar 24, 2022

Conversation

CoryMartin-NOAA
Copy link
Contributor

This PR adds:
ufsda python
ush/ufsda python scripts for creating YAML and other cool utilities. This name may change in the future, and I do expect eventually it to live outside of this repo (to share with RRFS, HAFS, 3DRTMA, etc.) but is fine here for now.

Note: for now, this is just some dummy cmake to copy the files over, we will want to install this as a python library eventually (possibly even as part of the pyvenv soon once things have stabilized).

sample j-jobs and ex-scripts
I've basically copied directly some sample j-jobs and ex-scripts from our old UFS-DA to here, purely as examples for @guillaumevernieres . They will not work as-is because of old path names, missing config, etc.

One other thing to note, would be the necessary changes needed to drive these from a high level script that would replace the configuration stored in global-workflow. All of these things (such as CDATE or GESDIR would need to be set by the driver/test script if not called by the workflow.

Another note: this builds off of PR #14

@CoryMartin-NOAA
Use new python env on orion
bd2d636
@CoryMartin-NOAA
Start of CI cron scripts
b9d6e10
@CoryMartin-NOAA
Add writing of comment message
14a2d1e
@CoryMartin-NOAA
Make it a login shell for modules
52a0c02
@CoryMartin-NOAA
Merge branch 'feature/new_orion_modules' into feature/orion_cron_rt
4a19577
@CoryMartin-NOAA
Fix typo
5d9a9ef
@CoryMartin-NOAA
Commit to save
d54c8c6
@CoryMartin-NOAA
move files to ci/
3135026
@CoryMartin-NOAA
Comment out repos for testing/debugging
3189c35
@CoryMartin-NOAA
Remove CRTM
4bfc5c7
@CoryMartin-NOAA
Add scrubber and change of labels
9f11f28
@CoryMartin-NOAA
Changes to the CI script
39dab94
@CoryMartin-NOAA
Add back all repos
86d99c2
@CoryMartin-NOAA
Update comment format
78484ef
@CoryMartin-NOAA
Move files from ufsda to GDASApp
7919d32
@CoryMartin-NOAA
Coding norms
9fae01e
@CoryMartin-NOAA
Merge branch 'develop' into feature/orion_cron_rt
6d84e38
@CoryMartin-NOAA
Add back unit tests on push
d78686c
@CoryMartin-NOAA
Changes to hera modules, and add hera ci config
a25b6a3
@CoryMartin-NOAA
Address reviewer comments round 1
2ff4158
@CoryMartin-NOAA
Merge branch 'feature/orion_cron_rt' into feature/hera_cron_rt
2559339
@CoryMartin-NOAA
Change test names
e1de501
@CoryMartin-NOAA
Remove ulimit
276e6d5
@CoryMartin-NOAA
Merge branch 'feature/orion_cron_rt' into feature/hera_cron_rt
5374a7a
@CoryMartin-NOAA
Merge branch 'develop' into feature/hera_cron_rt
b92580f
@CoryMartin-NOAA
j-jobs updated
56d30ea
@CoryMartin-NOAA
Merge branch 'develop' into feature/ufsda_files
248b7b9
@CoryMartin-NOAA
ex-scripts
a38f1af
@CoryMartin-NOAA
modify j-job
f97512e
@CoryMartin-NOAA
more changes
6c06b61
@emcbot emcbot added hera-RT-Running Automated testing running on Hera and removed hera-RT Queue for automated testing on Hera labels Mar 24, 2022
@CoryMartin-NOAA
Copy link
Contributor Author

ok @guillaumevernieres I propose we merge this in so you have the python tools and I will open a separate PR once I've solved the case of the disappearing test files.

@CoryMartin-NOAA CoryMartin-NOAA removed the orion-RT-Failed Automated testing failed on Orion label Mar 24, 2022
@emcbot
Copy link

emcbot commented Mar 24, 2022

Automated Pull Request Testing Results:
Machine: hera

Start: Thu Mar 24 16:30:16 UTC 2022 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Mar 24 17:25:17 UTC 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu Mar 24 17:25:36 UTC 2022
Tests: 33% tests passed, 2 tests failed out of 3
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/PR/19/GDASApp/build/log.ctest

@emcbot emcbot added hera-RT-Failed Automated testing failed on Hera and removed hera-RT-Running Automated testing running on Hera labels Mar 24, 2022
@CoryMartin-NOAA CoryMartin-NOAA added hera-RT Queue for automated testing on Hera orion-RT Queue for automated testing on Orion and removed hera-RT-Failed Automated testing failed on Hera labels Mar 24, 2022
@emcbot emcbot added orion-RT-Running Automated testing running on Orion hera-RT-Running Automated testing running on Hera and removed orion-RT Queue for automated testing on Orion hera-RT Queue for automated testing on Hera labels Mar 24, 2022
@emcbot
Copy link

emcbot commented Mar 24, 2022

Automated Pull Request Testing Results:
Machine: orion

Start: Thu Mar 24 13:30:14 CDT 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Mar 24 14:25:59 CDT 2022
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Thu Mar 24 14:26:24 CDT 2022
Tests: 100% tests passed, 0 tests failed out of 3

@emcbot emcbot added orion-RT-Passed Automated testing successful on Orion and removed orion-RT-Running Automated testing running on Orion labels Mar 24, 2022
@emcbot
Copy link

emcbot commented Mar 24, 2022

Automated Pull Request Testing Results:
Machine: hera

Start: Thu Mar 24 18:30:19 UTC 2022 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Mar 24 20:07:16 UTC 2022
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Thu Mar 24 20:07:26 UTC 2022
Tests: 100% tests passed, 0 tests failed out of 3

@emcbot emcbot added hera-RT-Passed Automated testing successful on Hera and removed hera-RT-Running Automated testing running on Hera labels Mar 24, 2022
@CoryMartin-NOAA CoryMartin-NOAA merged commit fa2611c into develop Mar 24, 2022
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/ufsda_files branch March 24, 2022 20:11
@CoryMartin-NOAA CoryMartin-NOAA mentioned this pull request Mar 25, 2022
Closed
guillaumevernieres added a commit that referenced this pull request May 28, 2024
@guillaumevernieres @CoryMartin-NOAA
Addin a GFSv17 ctest (#1130)
4598d91 
**What was done:**
- reorganization/code tidy to facilitate the addition of a GFSv17
prototype ctest
- exclude the new ctest by default, the user will have to run ```cmake
-DRUNGWCI=ON .``` on an old build to configure the new ctest

He're the current list:

```
 Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci
  Test  #1: WCDA-3DVAR-C48mx500
  Test  #2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200
  Test  #3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800
  Test  #4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800
  Test  #5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800
  Test  #6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800
  Test  #7: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800
  Test  #8: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800
  Test  #9: Aero-Snow-3DVAR-C96
  Test #10: Aero-Snow-3DVAR-C96_gdasfcst_202112201200
  Test #11: Atm-hyb-C96C48
  Test #12: Atm-hyb-C96C48_gdasfcst_202402231800
  Test #13: GFSv17-3DVAR-C384mx025
  Test #14: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000
  Test #15: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600
  Test #16: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600
  Test #17: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600
  Test #18: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600
  Test #19: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600
  Test #20: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600
  Test #21: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600
  Test #22: GFSv17-3DVAR-C384mx025_gdasprep_202106300600
  Test #23: GFSv17-3DVAR-C384mx025_gdasanal_202106300600

Total Tests: 23
```

---------

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
CoryMartin-NOAA pushed a commit that referenced this pull request Aug 6, 2024
@DavidNew-NOAA
Stage atmospheric backgrounds and UFS cubed-sphere history files (#1236)
f3fa26d 
This PR changes the GDASApp so that backgrounds for the atmospheric
analyses are staged as UFS cubed-sphere history files rather than FMS
restarts. jjob tests and test data are also updated to accommodate these
changes. The motivation being that FMS restarts include many data we
don't need for the analyses and thus take up too much disk
space/bandwidth.

This is a companion PR to Global Workflow
[#2792](NOAA-EMC/global-workflow#2792) and
JCB-GDAS [#19](NOAA-EMC/jcb-gdas#19).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guillaumevernieres guillaumevernieres guillaumevernieres approved these changes

@RussTreadon-NOAA RussTreadon-NOAA Awaiting requested review from RussTreadon-NOAA

@aerorahul aerorahul Awaiting requested review from aerorahul

Assignees

@CoryMartin-NOAA CoryMartin-NOAA

Labels
hera-RT-Passed Automated testing successful on Hera orion-RT-Passed Automated testing successful on Orion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@CoryMartin-NOAA @emcbot @guillaumevernieres