Fix masking of 3D MPAS-Ocean variables

[xylar]

Description

We have discovered that 3D ocean variables have not been correctly masked following #155. The review of that PR focused entirely on correctly masking sea-ice output and we missed the impact it had on 3D ocean variables, which was that is removed manual masking after remapping and assumed (incorrectly) that ncremap was handling the masking and renormaliztion itself. Instead, the 3D fields were being set to zero (not the _FillValue) below bathymetry, leading to fractional values after remapping, as reported in E3SM-Project/E3SM#6546

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

If applicable:

• New and existing unit tests pass with my changes (locally and CI/CD build)
• I have added tests that prove my fix is effective or that my feature works
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

[xylar]

CI seems to be failing for reasons unrelated to my one tiny change.

[chengzhuzhang]

It looks an issue with Codecov for CI to fail. @tomvothecoder could you please help with this?

[xylar]

@chengzhuzhang, the other thing that we may need to modify is that we are renormalizing with a threshold of 0.0 right now:
https://github.com/E3SM-Project/e3sm_to_cmip/blob/master/e3sm_to_cmip/mpas.py#L77
This may not be appropriate and we may end up with crazy temperature, salinity, etc.

In a quick test, the existing threshold of 0.0 seems to be okay at least for the depth slices I looked at.

[chengzhuzhang]

@xylar It looks like the threshold=0.0 only applies to sea-ice variables. I think based on some earlier experiment, using 0.0 worked okay for sea-ice. and to increase it to a higher value, e.g. 0.05 will mark more grids (5%) as missing.

[tomvothecoder]

It looks an issue with Codecov for CI to fail. @tomvothecoder could you please help with this?

Rebased with latest master including fix #271.

[chengzhuzhang]

Rebased with latest master including fix #271.

Thank you for a quick fix!

[xylar]

@xylar It looks like the threshold=0.0 only applies to sea-ice variables. I think based on some earlier experiment, using 0.0 worked okay for sea-ice. and to increase it to a higher value, e.g. 0.05 will mark more grids (5%) as missing.

@chengzhuzhang, all the calls I see to mpas.remap don't include the threshold flag, meaning they all default to 0.0. Could you clarify where the 0.05 value you refer to comes from? I don't see it.

[xylar]

I actually think it's the opposite of what you said: there's a 5% threshold for sea ice variables:

e3sm_to_cmip/e3sm_to_cmip/mpas.py

Line 235 in f816e64

def add_si_mask(ds, mask, siconc, threshold=0.05):

and 0.0 for the ocean variables (if they were properly masked).

Except that add_si_mask seems never to be used.

[chengzhuzhang]

We should remove the add_si_mask if it is not used.

Just to clarify, I only tested with threshold=0.0 and threshold=0.05(not implemented) in mpas.py when updating the sea-ice handlers.

The reason I said the threshold for re-nomalization only applies to sea-ice is based on the ncremap command, where no re-normalization flag (--rnr_thr) is applied to ocean variables:

e3sm_to_cmip/e3sm_to_cmip/mpas.py

Lines 99 to 119 in f816e64

	if pcode == "mpasocean":
	args = [
	"ncremap",
	"-P",
	f"{pcode}",
	"-7",
	"--dfl_lvl=1",
	"--no_stdin",
	"--no_cll_msr",
	"--no_frm_trm",
	"--no_permute",
	f"--map={mappingFileName}",
	inFileName,
	outFileName,
	]
	run_ncremap_cmd(args, env)
	elif pcode == "mpasseaice":
	# MPAS-Seaice is a special case because the of the time-varying SGS field
	remap_seaice_sgs(
	inFileName, outFileName, mappingFileName, renorm_threshold=threshold
	)

[xylar]

Oh, I see! That's another bug! And needs to be fixed here. But I won't be able to do it because I'm on vacation for the next 2 weeks.

[chengzhuzhang]

thanks for the heads-up, we can test and finalize this after you return. Enjoy the vacation!

[xylar]

@chengzhuzhang, I have got rid of the unused add_si_mask() function. I also made some changes so that there is a renormalization threshold of 5% for ocean variables. There is still a threshold of 0 for sea-ice variables. Let's see how this goes for some test data. If there are errors or obvious issues, please post here or on slack and I will see what I can do in the next 10 days. Otherwise, I'll take a careful look at the example output once I'm back.

[tomvothecoder]

Is this PR related to #255?

[xylar]

@tomvothecoder, I don't believe so.

[xylar]

@TonyB9000, any update on you having access to LCRC?

[TonyB9000]

@xylar I am logged into chrysalis (via the new "Duo Mobile" MFA). I recall having problems trying to fetch your changes for testing here (some permission issue). This could be my fault (git setup), but I believe I've pushed/pulled changes to my "datasm" codes from chrysalis before - or else I was hallucinating...

Just now, I issued:

(base) [ac.bartoletti1@chrlogin1 e3sm_to_cmip]$ git remote add xylar git@github.com:xylar/e3sm_to_cmip.git
error: remote xylar already exists.
(base) [ac.bartoletti1@chrlogin1 e3sm_to_cmip]$ git fetch xylar
Warning: Permanently added the ECDSA host key for IP address '140.82.113.4' to the list of known hosts.
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

So I am going to conduct the "Adding a new SSH key to your GitHub account" instructions, to see if I can get that straightened out.

[xylar]

@TonyB9000, yes, it sounds like your ssh keys aren't set up right for LCRC on GitHub.

[TonyB9000]

@xylar It seems I have the keypair

-rw-------  1 ac.bartoletti1 cels  444 Aug 15 19:32 id_ed25519
-rw-r--r--  1 ac.bartoletti1 cels   94 Aug 15 19:32 id_ed25519.pub

And I just added it to my git profile - but I have no idea what to enter for a passphrase (I tried my acme1 keyphrase and it just keeps asking . . . I guess I need to generate a new one.

[xylar]

Yes, generate a new one. It's a pretty big pain to have a password because you have to enter it dozens of times if you do a recursive checkout. I personally think it's safe to generate a new one without a password.

[TonyB9000]

@xylar OK, I can now fetch and see (on chrysalis):

From github.com:xylar/e3sm_to_cmip
 * [new branch]      fix-mpas-fill-values            -> xylar/fix-mpas-fill-values
 * [new branch]      fix-mpaso-3d-masking            -> xylar/fix-mpaso-3d-masking
 * [new branch]      master                          -> xylar/master
 * [new branch]      move_scripts_cwl_into_package   -> xylar/move_scripts_cwl_into_package
 * [new branch]      separate_ocean_and_seaice_masks -> xylar/separate_ocean_and_seaice_masks

How should I proceed from here? I assume I want to create a conda env and "pip install" something...

[xylar]

@TonyB9000, the branch name is at the top of the PR here, fix-mpaso-3d-masking. I'm afraid I don't have time to teach to tools of the trade right now (it's 10 at night where I live). Maybe @chengzhuzhang can help?

[chengzhuzhang]

@TonyB9000 I'm copying the steps from Slack DMs:

git remote add xylar git@github.com:xylar/e3sm_to_cmip.git
git fetch xylar
git switch fix-mpaso-3d-masking

Then you can install to the environment that you intend to test this new feature.

[TonyB9000]

@chengzhuzhang Thanks Jill. I hope there is also a way to verify the presence/absence of the problem. We have 400 datasets we need to regenerate - I believe. I can produce that list.

I am a bit concerned when I see

* [new branch] master -> xylar/master

But "git branch" lists

  consolidate_cmor_setups_and_logging
* master

I have abandoned the branch "consolidate_cmor_setups_and_logging" in favor of two branches, "consolidate_cmor_setups_273" and "redesign_logger_274". After Tom's review, I merged "consolidate_cmor_setups_273", then discovered I had missed a few stragglers, and opened/pushed/PR's "missed_cmor_consolidations".

On chrysalis, do I need to "fetch" those other branches? And how would I separate my "master" from xylar's?

[chengzhuzhang]

@TonyB9000 I'm not sure if I understand your question. But for testing this PR, you can just install this branch fix-mpaso-3d-masking. After provide some example output, we should request review from Xylar. Merging this branch with other changes on master can be the next step.

[TonyB9000]

@chengzhuzhang OK, I created the env "dsm_test_xylar_e2c", pip installed both datasm (so I can access my auxiliary tools).

Then I cd'd to my gitrepo/e3sm_to_cmip, did "git checkout" of Xylar's branch:

(dsm_test_xylar_e2c) [ac.bartoletti1@chrlogin1 e3sm_to_cmip]$ git checkout fix-mpaso-3d-masking
branch 'fix-mpaso-3d-masking' set up to track 'xylar/fix-mpaso-3d-masking'.
Switched to a new branch 'fix-mpaso-3d-masking'

Then I pip-installed this branch.

I can now run the "end-to-end" script (Operations/5_DatasetGeneration/Alt_Process/dsm_generate_CMIP6.sh) on some selected set of v2 and v2_1 datasets - where I happen to have the corresponding NATIVE data in the warehouse. I'll have to see what native v2 data exists.

My real question is, when I decide to switch back to my master branch, will I get my original master, or the one that reads

• [new branch] master -> xylar/master

(BTW: Turns out I only have v2_1 and some v3 native data, and of the v2_1 only 1pctCO2 and amip. The amip won't do us any good here (no MPAS) so I will need to test/generate some v2_1/1pctCO2/ocean stuff. I suppose a single year is sufficient for testing. But we should copy over some v2 and v2-NARRM as well, I suppose.)

[TonyB9000]

@chengzhuzhang These 10 cmip6 sets:

CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.hfsifrazil.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.masscello.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.so.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.thetao.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.thkcello.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.uo.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.vo.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.volcello.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.wo.gr
CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.zhalfo.gr

All rely upon the one native set I have on hand:

E3SM.2_1.1pctCO2.LR.ocean.native.model-output.mon.ens1

So I will run a 1-year test for each and we can examine the results.

[TonyB9000]

@xylar @chengzhuzhang We have completed 1 year each of the above 10 v2_1 1pctCO2 Omon CMIP6 datasets. The output files are in (chrysalis):

/lcrc/group/e3sm2/ac.bartoletti1/Operations/5_DatasetGeneration/AltProcess/tmp/v2_1.LR.1pctCO2_0101/product/CMIP6/CMIP/E3SM-Project/E3SM-2-1/1pctCO2/r1i1p1f1/Omon/*/gr/v20240909/

where "*" is any one of the 10 Omon variables listed above.

The only unusual log output was for hfsifrazil, with cmor warning

Warning: variable 'hfsifrazil' (table Omon) you passed positive value:down, but table does not mention it, will be ignored, if you really want this in your variable output use cmor_set_variable_attribute_internal function.

NOTE: As this version of e2c has not had the logfile consolidation work applied, 8 of the cmor logs are in

/lcrc/group/e3sm2/ac.bartoletti1/Operations/5_DatasetGeneration/AltProcess/cmor_logs

Two are in

/lcrc/group/e3sm2/ac.bartoletti1/Operations/5_DatasetGeneration/AltProcess/logs

The more verbose e2c logs are in

/lcrc/group/e3sm2/ac.bartoletti1/Operations/5_DatasetGeneration/AltProcess/tmp/v2_1.LR.1pctCO2_0101/caselogs (the ones that end with ".sublog")

[xylar]

@TonyB9000, the temp file looks like what I was hoping for so I think you did things right. Unfortunately, it didn't fix the problem.

[TonyB9000]

@xylar (Yes, I did the cherry-pick thing - forgot the details). Sorry it didn't work.

Let me know what else I can do, as soon as you're ready.

[xylar]

@TonyB9000, could you try one more time with the thetao workflow after cherry-picking my latest commit 33a25f8?

[xylar]

If that doesn't work, I'm going to need to ask Charlie for some help.

[TonyB9000]

@xylar Done. Files should be readable:

infile = /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmppmiocw24
outfile = /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpj3gdmqec

[xylar]

Thanks @TonyB9000! That also did what I wanted but didn't fix the problem. I'm going to experiment some more over the weekend and I'll ask for help from you or Charlie or whoever once I have a better idea what could be going on.

[TonyB9000]

@xylar Charlie may know - I an clueless egarding cmor/xarray internals... Cheers!

[xylar]

Okay, I think I've finally got it.

thetao:

uo:

[xylar]

The problem turned out to be that I was keeping the _FillValue attribute if it existed when writing out the NetCDF file from xarray. But this seems to mean thtat xarray doesn't change invalid values from NaN to the fill value as part of writing out the data. It is simpler to just drop existing _FillValue attributes and update the fill value to the NetCDF default value during output.

In any case CMOR then changes the fill value to its preferred value down the road.

The issue with NaNs is that aren't fill values (and maybe even using NaNs as the fill value) is that NCO then tries to renormalize them and gets NaNs everywhere that even overlaps a masked source cell.

[xylar]

@TonyB9000, can you do the following?

git fetch --all
git reset --hard xylar/fix-mpaso-e3-masking
python -m pip install .
...

I'm assuming your name for my remote is xylar but it could be something else. Use git remote -v to find out what you called it.

Could you then rerun the workflow on all variables as before? I want to make very sure the fix is working before we redo a bunch of ocean 3D variables.

[TonyB9000]

@xylar Great! I'll work on that this morning.

[TonyB9000]

@xylar Minor git issue: The first command went well:

Note:  git branch
(base) [ac.bartoletti1@chrlogin2 e3sm_to_cmip]$ git branch
* fix-mpaso-3d-masking
  master

Then

(base) [ac.bartoletti1@chrlogin2 e3sm_to_cmip]$ git fetch --all
Fetching origin
Fetching xylar
Enter passphrase for key '/home/ac.bartoletti1/.ssh/id_ed25519':
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 8 (delta 6), reused 8 (delta 6), pack-reused 0 (from 0)
Unpacking objects: 100% (8/8), 895 bytes | 24.00 KiB/s, done.
From github.com:xylar/e3sm_to_cmip
 + 33a25f8...e332f93 fix-mpaso-3d-masking -> xylar/fix-mpaso-3d-masking  (forced update)

The second command complained:

(base) [ac.bartoletti1@chrlogin2 e3sm_to_cmip]$ git reset --hard xylar/fix-mpaso-e3-masking
fatal: ambiguous argument 'xylar/fix-mpaso-e3-masking': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Trying git remote -v was not helpful:

(base) [ac.bartoletti1@chrlogin2 e3sm_to_cmip]$ git remote -v
origin  https://github.com/E3SM-Project/e3sm_to_cmip (fetch)
origin  https://github.com/E3SM-Project/e3sm_to_cmip (push)
xylar   git@github.com:xylar/e3sm_to_cmip.git (fetch)
xylar   git@github.com:xylar/e3sm_to_cmip.git (push)

Sorry for being a gitiot...

[TonyB9000]

@xylar Ahh I got it. Type I guess I want "3d-masking" not "e3-masking".

[xylar]

Yes, just a typo. I just saw that, too.

[TonyB9000]

@xylar So far, the only complaint was this one that returned (from cmor) for hfsifrazil)

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: variable 'hfsifrazil' (table Omon) you passed positive value:down, but table does not mention it, will be ignored, if you really want this in your variable output use cmor_set_variable_attribute_internal function
!
!!!!!!!!!!!!!!!!!!!!!!!!!

(I am not fond of the cmor log-output format.)

Test runs are completed. The e2c logs are in

/lcrc/group/e3sm2/ac.bartoletti1/Operations/5_DatasetGeneration/AltProcess/tmp/v2_1.LR.1pctCO2_0101/caselogs

and grep Retaining 20240923*.sublog | fgrep "[INFO]" | sort | uniq | cut -f2- -d: yields these 20 temp-files:

20240923_160307_324873-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.hfsifrazil.gr.sublog:2024-09-23 16:03:29,385 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpx6glycxe
20240923_160307_324873-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.hfsifrazil.gr.sublog:2024-09-23 16:03:29,386 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpwy2dr3us
20240923_160334_982650-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.masscello.gr.sublog:2024-09-23 16:03:53,871 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp1en1fr3n
20240923_160334_982650-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.masscello.gr.sublog:2024-09-23 16:03:53,872 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpl2bmk8md
20240923_160400_834344-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.so.gr.sublog:2024-09-23 16:04:21,177 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpkdo7yd5z
20240923_160400_834344-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.so.gr.sublog:2024-09-23 16:04:21,177 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmprutxxcdf
20240923_160428_672629-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.thetao.gr.sublog:2024-09-23 16:04:50,396 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpavvl9sgu
20240923_160428_672629-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.thetao.gr.sublog:2024-09-23 16:04:50,396 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp8lhyvsv9
20240923_160458_979829-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.thkcello.gr.sublog:2024-09-23 16:05:17,273 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpo5tyuwlq
20240923_160458_979829-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.thkcello.gr.sublog:2024-09-23 16:05:17,273 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpwqjsogzr
20240923_160524_660809-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.uo.gr.sublog:2024-09-23 16:05:44,358 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp3vdpkpfu
20240923_160524_660809-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.uo.gr.sublog:2024-09-23 16:05:44,358 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp31lyopo8
20240923_160554_167934-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.vo.gr.sublog:2024-09-23 16:06:15,329 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpulkgvf1y
20240923_160554_167934-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.vo.gr.sublog:2024-09-23 16:06:15,330 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpdsdju7bx
20240923_160625_512799-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.volcello.gr.sublog:2024-09-23 16:06:41,898 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp6v25hqdk
20240923_160625_512799-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.volcello.gr.sublog:2024-09-23 16:06:41,898 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpmnwy9adp
20240923_160649_548878-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.wo.gr.sublog:2024-09-23 16:07:12,738 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp0tbghyfa
20240923_160649_548878-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.wo.gr.sublog:2024-09-23 16:07:12,738 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp0kfcdkht
20240923_160721_095861-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.zhalfo.gr.sublog:2024-09-23 16:09:31,483 [INFO]: mpas.py(remap:146) >> Retaining inFileName  /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpl4a0eh0a
20240923_160721_095861-CMIP6.CMIP.E3SM-Project.E3SM-2-1.1pctCO2.r1i1p1f1.Omon.zhalfo.gr.sublog:2024-09-23 16:09:31,483 [INFO]: mpas.py(remap:147) >> Retaining outFileName /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpnqj5py31

(permissions updated.)

[xylar]

Thanks @TonyB9000!

Here are the plots from the latest processed files:

Everything looks great to me! I think this is ready to merge unless there are further concerns. (We don't want to include the changes to keep the temp files. That was just for debugging.)

[TonyB9000]

@xylar This is great! I'll perform the merge and (eventually) rebase my other branches.

I can simply change the variable "retain_temp_files" to "False" in the code before merging. That way, future testing need not recall how to log the names of the given tempfiles.

[xylar]

Thanks @TonyB9000! Now the hard work begins. Let me know if I can be of help there.

[TonyB9000]

@xylar "Hard work for the processors". For me, almost everything is automated. I take the list of 400 v2/v2_1 Ocean 3D-var CMIP6 dataset_ids (already broken down to 80 v2_1, 320 v2), and feed them to "dsm_generate_CMIP6.sh", changing "TEST" to "WORK" so that all the years are generated and the outputs are moved to the warehouse in preparation for publication (and the dataset status files are automatically updated). It may take a week or more to run, but all in background.

Meanwhile, we have the other 917 v2_1 CMIP6 datasets to publish, and we have just now "established" (not yet tested) the new methodology for publishing E3SM datasets from ANL/chrysalis, So everything is lining up OK.

[xylar]

I'm glad it's not an overwhelming amount of human time, at least!

[TonyB9000]

@xylar (at your leisure) Could you check one final "test set" of thetao:

infile = /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmp7n6ee9w5
outfile = /lcrc/group/e3sm2/ac.bartoletti1/tmp/tmpzjpoorha

I rebased my branch of "redesign_logger_274" to reflect your changes to ocean masking (merged to master), and then ran a test with the revised logging. I want to ensure I have not screwed anything up...

[xylar]

@TonyB9000, yep the masking still looks good in tmpzjpoorha, so I think we're good.

[TonyB9000]

@xylar Awesome! Thanks.