Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The staging of the GFSv17 prototype IC's fails #2890

Closed
guillaumevernieres opened this issue Sep 4, 2024 · 11 comments · Fixed by #3215
Closed

The staging of the GFSv17 prototype IC's fails #2890

guillaumevernieres opened this issue Sep 4, 2024 · 11 comments · Fixed by #3215
Assignees
@KateFriedman-NOAA
Labels
bug Something isn't working

Comments

@guillaumevernieres
Copy link
Contributor

What is wrong?

The "refactored" staging of the IC's for the C384/025 GFSv17 prototype does not work anymore.

What should have happened?

...

What machines are impacted?

All or N/A

Steps to reproduce

Run this: https://github.com/NOAA-EMC/global-workflow/blob/develop/ci/cases/gfsv17/C384mx025_3DVarAOWCDA.yaml

Additional information

...

Do you have a proposed solution?

No response
@guillaumevernieres guillaumevernieres added bug Something isn't working triage Issues that are triage labels Sep 4, 2024
@KateFriedman-NOAA KateFriedman-NOAA removed the triage Issues that are triage label Sep 4, 2024
@KateFriedman-NOAA
Copy link
Member

My guess is that it no longer is able to stage because of the model_data -> model subfolder rename. The ICSDIR in that yaml (/scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/IC_2021063000_V2) has model_data folders that need to be renamed to model. I do not have access to do that. Would need @JessicaMeixner-NOAA to rename.

@guillaumevernieres can you provide a log with the failure so I can check that that's the issue? Thanks!

@guillaumevernieres
Copy link
Contributor Author

@KateFriedman-NOAA , the failure isn't related to the renaming. Mindo made a copy of @JessicaMeixner-NOAA 's IC's and renamed to model but it failed grabbing the correct restart. I'll ask Mindo to point you to his log.

@KateFriedman-NOAA
Copy link
Member

@guillaumevernieres I see, thanks for that additional context. I'll take a look at the log when shared and see what needs fixing.

@apchoiCMD
Copy link

apchoiCMD commented Sep 5, 2024 •
edited
Loading

@KateFriedman-NOAA Here is the log and thanks!

/scratch1/NCEPDEV/da/Mindo.Choi/workflow_08272024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/GFSv17-3DVAR-C384mx025/COMROOT/GFSv17-3DVAR-C384mx025/logs/2021063000/gdasstage_ic.log

@KateFriedman-NOAA
Copy link
Member

@KateFriedman-NOAA Here is the log and thanks!

workflow_08272024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/GFSv17-3DVAR-C384mx025/COMROOT/GFSv17-3DVAR-C384mx025/logs/2021063000/gdasstage_ic.log

@apchoiCMD I think the first part of the log path got cut off. Can you correct it? Thanks!

@apchoiCMD
Copy link

@KateFriedman-NOAA Here is the log and thanks!
workflow_08272024/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci/GFSv17-3DVAR-C384mx025/COMROOT/GFSv17-3DVAR-C384mx025/logs/2021063000/gdasstage_ic.log

@apchoiCMD I think the first part of the log path got cut off. Can you correct it? Thanks!

Sorry for the confusion- made a correction.

@KateFriedman-NOAA
Copy link
Member

Thanks for the log @apchoiCMD ! I see what's happening and I believe it's a similar issue to issue #2865. I will try to run the same test and work on a fix.

@KateFriedman-NOAA KateFriedman-NOAA mentioned this issue Sep 6, 2024
Closed
@KateFriedman-NOAA
Copy link
Member

KateFriedman-NOAA commented Sep 6, 2024 •
edited
Loading

@apchoiCMD @guillaumevernieres I believe I have fixed this. Would you mind testing the fix? Please copy the staging yaml files from my clone on Hera (/scratch1/NCEPDEV/global/Kate.Friedman/git/develop_fork/parm/stage) into your clone's parm/stage folder and try your case again. Let me know how it goes, thanks!

@apchoiCMD
Copy link

@KateFriedman-NOAA Thanks for your fix- From my side, the test is successful-

image

@KateFriedman-NOAA
Copy link
Member

Thanks for testing and confirming @apchoiCMD !

@guillaumevernieres
Copy link
Contributor Author

Thanks for testing and confirming @apchoiCMD !

And thanks for the quick fix @KateFriedman-NOAA .

KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Sep 12, 2024
@KateFriedman-NOAA
Differentiate component start date variables
362ae92 
- Create specific variables for atmos, ice, mediator, ocean,
and wave cycle dates and update scripts/staging yamls to use them.
- Also resolve issue with incorrect IC filenames caused by not
turning IAU off for cold-start.

Refs NOAA-EMC#2865
Refs NOAA-EMC#2890
@KateFriedman-NOAA KateFriedman-NOAA mentioned this issue Sep 12, 2024
Closed
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this issue Jan 8, 2025
@KateFriedman-NOAA
Turn off IAU in staging job when cold started
cc83660 
Fixes issue with incorrectly setting model_start time
for initial condition filenames when cold starting.

Refs NOAA-EMC#2865
Refs NOAA-EMC#2890
@KateFriedman-NOAA KateFriedman-NOAA mentioned this issue Jan 8, 2025
Merged
3 tasks
KateFriedman-NOAA added a commit that referenced this issue Jan 14, 2025
@KateFriedman-NOAA
Turn IAU off during staging job for cold start experiments (#3215)
53ed76e 
Resolve a bug related to IAU when cold-starting an experiment.

The model start time for non-atmos ICs were incorrectly being reduced by 3hrs
for cold-started experiments. While a user may want IAU on (`DOIAU=YES`), the
IAU switch should be set to "NO" for the staging job when it is cold-started.

Resolves #2890
Resolves #2865
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KateFriedman-NOAA KateFriedman-NOAA

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
3 participants
@guillaumevernieres @KateFriedman-NOAA @apchoiCMD