Flip atmos/memXXX to memXXX/atmos

[aerorahul]

Description

This PR:

• flips the ensemble member directories which are subdirectories under components to having all the components for any member under the member directory.
• Fixes Correct the hierarchy of ensemble members in ROTDIR #1196
• The "atmos" jobs define "COMPONENT=atmos". They are "atmos" jobs, so there is no need to define a variable "COMPONENT" as it is obvious. Removes "${COMPONENT}" in the "atmos" jobs and make it explicit.

This PR also adds a "hack" to allow the success for the gdaspostanl job in the first half cycle.
The "hack" will be removed with the refactoring of the post jobs in the (near) future.

This change is necessary as GFSv17 will introduce coupled model components in the ensemble for initialization of the GEFS. Copying ICs for GEFS from GFS will be greatly simplified and the information for any given member will be self contained.

Since the changes in this PR are exhaustive, the linter recommendations will be addressed in a separate PR.

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

• A cycled experiment on Hera at C96/C48 ensemble has been conducted. Output can be found at:

EXPDIR: /work2/noaa/stmp/rmahajan/gwWork/EXPDIR/tmem01
ROTDIR: /work2/noaa/stmp/rmahajan/gwWork/ROTDIR/tmem01

• A cycled experiment on Orion was also started and compared w/ develop. See this comment.
• The hpssarch_gen.sh script is quite confusing and at this time, the archival of the ensemble data has been updated but has not been verified. Further PR's will address the issue. The earc step does succeed without errors. Since this was tested on Orion, it is unclear if this actually works on Orion in the first place. A cycled test on Hera/WCOSS2 might be necessary.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

requesting re-review.

[aerorahul]

Adding @XianwuXue-NOAA to ensure the early cycle EnKF does not run into issues with this branch.

[XianwuXue-NOAA]

Tested early-EnKF successfully after the change.

[aerorahul]

@CatherineThomas-NOAA @RussTreadon-NOAA
We would appreciate a review from either of you before we merge this PR.

[RussTreadon-NOAA]

If the changes have been confirmed to not alter results in a cycled test of sufficient duration, the changes should be OK. If I were to review the changes this is what I would do. I would run at least two parallels. One would use g-w develop. The other would use aerorahul:feature/flip-memdir. Time and resources permitting I would also see if the changes in aerorahul:feature/flip-memdir reproduce the operational GFS.

[RussTreadon-NOAA]

Please see my comment #1201 (comment) for my feedback regarding these changes. Given confirmation that parallel test(s) have been run using aerorahul:feature/flip-memdir for sufficient duration with no change in cycled results, the changes in this PR should be merged into develop.

[aerorahul]

@RussTreadon-NOAA
Thank you for your thoughtful comments.

Two experiments have been started on Orion:

control: /work2/noaa/stmp/rmahajan/gwWork/EXPDIR/devmem
experiment: /work2/noaa/stmp/rmahajan/gwWork/EXPDIR/expmem

The control uses develop and the experiment uses the branch from this PR.
The experiments are set to run for 4.5 cycles with GLDAS and MET OFF.
If this is not enough, then please let me know.

Running operational tests is not feasible as develop has significantly evolved from the operational tag.

I will do comparisons in the directories every cycle randomly for different members and report them here.

I will also attempt comparing HPSS tarballs. Though, that can be deferred to later PR as we are in the process of cataloging those files differently.

[RussTreadon-NOAA]

The devmem and expmem parallels are still running on Orion. A spot check of files for 20211221/00 finds consistency between the two experiments. This is a good sign.

[aerorahul]

A few spot checks:

❯❯❯ date=2021122100; mem=013; file=gdas.t${date:8:2}z.atminc.nc; cmp devmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/atmos/mem$mem/$file expmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/mem$mem/atmos/$file
❯❯❯

❯❯❯ date=2021122106; mem=008; file=gdas.t${date:8:2}z.atmf006.nc; cmp devmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/atmos/mem$mem/$file expmem/ROTDIR/enkfgdas.${date:0:8}/${date:8:2}/mem$mem/atmos/$file
❯❯❯

[RussTreadon-NOAA]

Orion parallels out to 2021122118. Spot check of 12Z cycle looks good. Give tentative approval. Will check more cycles later.

[RussTreadon-NOAA]

Spot check of gfs, gdas, and enkfgdas at 20211222/00 shows consistency between the control devmem and the test expmem. Some files differ but these are expected and acceptable differences (e.g., master grib index files, loginc, loganl, gsi diagnostic files).

I discovered a new file, input.nml in gfs.$PDY/$cyc/atmos. It's in both devmem and expmem. forecast_postdet.sh copies this file to COMOUT when gfsfcst runs.

[CatherineThomas-NOAA]

Thanks for running the tests. Results look good to me as do the code changes themselves. Approve with the understanding that two aspects will be addressed in future PRs:

• archival
• top level non-member directory (enfkstat, spread, etc)