Recommend a directory structure for input/intermediate/output files

[bitterbark]

It came up in the walk-through that we might want to suggest a standard for directory structures in the config file.
Having a standard would be nice right now to keep from having a mess of dirs and files; in the future it will help us find each other's stuff easily.

Note that the figures currently are written to $cam_diag_plot_loc/$case_vs_$baseline_XXX/website.
And the regrid location is currently given outside of the case specifications, "used by all diagnostic runs"

@andrewgettelman (who raised the issue on the discussion) & others who ran the package ( @islasimpson , @phillips-ad , @cecilehannay, others?) do you want to suggest what you found useful?

[andrewgettelman]

Assuming stuff gets put on scratch (e.g. /glade/scratch/user/ADF) and the code itself for ADF scripts is in home or work directories (not purged)....

One approach might be ./ADF/diag and ./ADF/climo where $diag_plot_loc = ./ADF/diag and the regid and climatologies are listed under ./ADF/climo by casename. Then other runs against different (or eventually multi) cases would end up in something like ./ADF/diag/$case_vs_$baseline_XXX/website

I think that is similar to how I have done things in the past. Happy to defer to others' workflows, especially @cecilehannay. I don't have a strong opinion

[phillips-ad]

I second the directory naming/structure that @andrewgettelman proposed, and would also defer to @cecilehannay. Not directory structure related per se, but as previously mentioned I also think the climo output file names should contain the years the calculation is based on. Following Andrew's proposed directory structure, a sample file could look like this:
./ADF/climo/b.e21.BSSP245smbb.f09_g17.008/b.e21.BSSP245smbb.f09_g17.008_SWCF_climo.2020-2030.nc

I can open a new discussion/issue on the climo file naming topic if needed; I wasn't sure whether to piggyback on this thread or not.

[dan800]

I would suggest not trying to use the directory name as a placeholder for information on the contents of the diagnostic output. It doesn't work when multiple cases are being compared. It leads to incredibly long paths - e.g.,
ADF/plots/f.e21.FWscHIST.ne30_L48_BL10_cam6_3_041_kzz3_zmtop75.hf.001/ADF/f.e21.FWscHIST.ne30_L48_BL10_cam6_3_041_kzz3_zmtop75.hf.001_vs_f.e21.FWscHIST.ne30_L48_BL10_cam6_3_041_control.hf.001_1980_1989/assets/
A more radical approach that doesn't constrain the contents of the output would be to simply tag the directory with the something like ADF-date-UT where date-UT is the time the diags were created. At the top level a file could be included (xml or json) that includes all the metadata for the particular run of the ADF diagnostics. This could include the years used in each case, the various cases included and the reference case if used. We already have tables that list what the cases are and links to the diags - the link would just a lot smaller - e.g., ADF/ADF-20220310-1442/plots/assets

[andrewgettelman]

I agree that the names can get long for two casenames.

But I would be scared of your radical approach. We would end up with lots and lots of directories with inexplicable names.

Maybe as an alternative, we could allow the user to specify another name instead of the case name? E.g.: case1dirname = 'Tst2', case2dirname='control'.

We would then end up with: ADF/plots/Tst2_vs_control/assets

I think we already have a title name for the case in the plots (but that could have spaces in it).

Regards,

Andrew

[dan800]

I still think this approach doesn't work for ensembles or even a three way comparison. The idea is you don't look at the directory name to determine the case, model configuration and period (e.g., year 300-350 vs year 0-50 of a PI spin up) - you get to it via the issue page under the Diagnostics bullet:
e.g., NCAR/amwg_dev#83
or some table that lists the various runs and has a column for the diag links which could also show a lot more info than is in the directory name.
Anyway, just a suggestion.