Toolchain part2: `./mkmap_data.py` #1430

negin513 · 2021-07-16T16:19:23Z

Description of changes

This Python script is the second part of the simplified toolchain for creating surface datasets for global ctsm cases (#644 ).

After creating the namelist/control file with ./gen_mksurf_namelist.py (PR #1419) using the options for your desired case, you should run :

./mkmap_data.py --namelist [namelist from ./gen_mksurf_namelist.py]

to create mapping/weight files.

This python script is the alternative code to mkmapdata.sh with few differences:

It reads namelist file for DST mesh file.
For raw dataset in the namelist it finds their SRC mesh file and mask from
the necdf metadata.
Check if the weight (mapping file) already exists or not.
If it does not exist, it creates the mapping (weight) file.

Specific notes

Contributors other than yourself, if any: @slevisconsulting

CTSM Issues Fixed (include github issue #):

Are answers expected to change (and if so in what way)?

Any User Interface Changes (namelist or namelist defaults changes)?

./mkmap_data.py has the following interface:

usage: mkmap_data.py [-h] --namelist NAMELIST [--input_dir INPUT_DIR]
                     [--map_dir MAP_DIR] [-d] [--overwrite]
                     [--esmf_path OVERWRITE] [--account ACCOUNT_NAME]

|------------------------------------------------------------------|
|---------------------  Instructions  -----------------------------|
|------------------------------------------------------------------|
This Python script is part of the simplified toolchain for creating
the surface dataset for ctsm cases.

After creating the namelist/control file with ./gen_mksurf_namelist.py
using the options for your desired case, you should run :

./mkmap_data.py --namelist [namelist from ./gen_mksurf_namelist.py]

to create mapping files.

This python script is the alternative code to mkmapdata.sh with
 few differences:

1. It reads namelist file for DST mesh file.

2. For raw dataset in the namelist it finds their SRC mesh file and mask from
the necdf metadata.

3. Check if the weight (mapping file) already exists or not. 

4. If it does not exist, it creates the mapping (weight) file. 
 
-------------------------------------------------------------------
Instructions for running on Cheyenne/Casper:
 
load the following into your local environment:
 
    module load python
    ncar_pylib
-------------------------------------------------------------------
To see the available options:
    ./mkmap_data.py --help
 
To run the script:
    ./mkmap_data.py --namelist ${namelist from ./gen_mksurf_namelist.py}
 
To remove NPL from your environment on Cheyenne/Casper:
    deactivate
-------------------------------------------------------------------

optional arguments:
  -h, --help            show this help message and exit
  --namelist NAMELIST   namelist for the ctsm case created by
                        gen_mksurf_namelist.py
  --input_dir INPUT_DIR
                        Path of your mesh files and input data.', [default:
                        /glade/p/cesm/cseg/inputdata/]
  --map_dir MAP_DIR     Path of your weight (mapping) files.', [default: /glad
                        e/scratch/negins/ctsm_toolchain_2/tools/toolchain]
  -d, --debug           Debug mode will print more information [default:
                        False]
  --overwrite           Flag to overwrite the existing mapping files.
                        [default: False]
  --esmf_path OVERWRITE
                        Path of the ESMF library. [default: None]
  --account ACCOUNT_NAME
                        Cheyenne or Casper project account name.', [default:
                        P93300606]

Testing performed, if any: @slevisconsulting

slevis-lmwg

@negin513 thank you for your nice work on the toolchain! Also, thank you for giving me a tour of the PR this morning.

I have a very important general comment that I realized after we spoke:
According to the design that the group adopted (slide 5 in this presentation), this piece of code should also generate the "old" namelist (the one that the fortran code reads). The "old" namelist has what the new namelist contains PLUS the list of map_ files. I think that's all.

slevis-lmwg · 2021-07-16T20:36:38Z

tools/toolchain/mkmap_data.py

+|------------------------------------------------------------------|
+|---------------------  Instructions  -----------------------------|
+|------------------------------------------------------------------|
+This Python script is part of the simplified toolchain for creating


Suggested change

This Python script is part of the simplified toolchain for creating

This Python script is STEP 2 of the simplified toolchain for creating

slevis-lmwg · 2021-07-16T20:40:29Z

tools/toolchain/mkmap_data.py

+
+3. Check if the weight (mapping file) already exists or not. 
+
+4. If it does not exist, it creates the mapping (weight) file. 


Steps 3 and 4 are not different from mkmapdata.sh. For this documentation to be clear, you should add at the end of 3 and 4 something in parentheses that says (same as mkmapdata.sh).

slevis-lmwg · 2021-07-16T20:41:43Z

tools/toolchain/mkmap_data.py

+To run the script:
+    ./mkmap_data.py --namelist ${namelist from ./gen_mksurf_namelist.py}
+
+To remove NPL from your environment on Cheyenne/Casper:


Suggested change

To remove NPL from your environment on Cheyenne/Casper:

To remove NPL (ncar_pylib) from your environment on Cheyenne/Casper:

slevis-lmwg · 2021-07-16T20:45:44Z

tools/toolchain/mkmap_data.py

+from datetime import datetime
+from getpass import getuser
+
+# from gen_mksurf_namelist import get_parser, build_nl


is this a placeholder or can be deleted?

is this a placeholder to address the generation of the "old" namelist?

slevis-lmwg · 2021-07-16T20:47:41Z

tools/toolchain/mkmap_data.py

+
+    parser.add_argument(
+        "--namelist",
+        help="namelist for the ctsm case created by gen_mksurf_namelist.py ",


Suggested change

help="namelist for the ctsm case created by gen_mksurf_namelist.py ",

help="namelist created by gen_mksurf_namelist.py ",

slevis-lmwg · 2021-07-16T22:28:59Z

tools/toolchain/mkmap_data.py

+        if "SCRIP" in self.mesh_file:
+            self.mesh_type = "SCRIP"
+        # else:
+        # TODO: TALK to SAM about thi


This function may be going away, as discussed elsewhere.

slevis-lmwg · 2021-07-16T22:30:29Z

tools/toolchain/mkmap_data.py

+    logging.debug("\n dst_mesh  :")
+    logging.debug(dst_mesh)
+
+    # -- src mesh files


Suggested change

# -- src mesh files

# -- raw dataset files containing corresponding src mesh file names

slevis-lmwg · 2021-07-16T22:41:52Z

tools/toolchain/mkmap_data.py

+    logging.debug(dst_mesh)
+
+    # -- src mesh files
+    src_flist = [


How about cycling through everything in the namelist with the prefix "mksrf_"
instead of hardwiring this whole list?

slevis-lmwg · 2021-07-16T22:47:17Z

tools/toolchain/mkmap_data.py

+        logging.debug("\n" + src_file + " : " + src_fname)
+
+        # -- check if src_file exist
+        if not os.path.isfile(src_fname):


I think you already do this in class MeshType def __init__ so you could delete this if section and keep the else.

tools/toolchain/mkmap_data.py

+
+    mapping_fname: str
+        Mapping /weight filename, if not explicitly given the
+        class will create one based on src_mesh and dst_mesh information.


slevis-lmwg · 2021-07-23T17:33:35Z

According to the design that the group adopted (slide 5 in this presentation), this piece of code should also generate the "old" namelist (the one that the fortran code reads). The "old" namelist has what the new namelist contains PLUS the list of map_ files. I think that's all.

@negin513 recommended that it would be cleaner to put the function of creating the old namelist in a separate script that we call gen_old_namelist.py. See updated slides (same link). Also gen_old_namelist.py will be a step that's transparent to the user and will be run by the wrapper script. The wrapper script will be a new pull request (part 3 in this series of PRs).

ekluzek · 2022-02-28T23:27:36Z

With the work on the parallel mksurfdata_map, this PR will likely be closed without coming in.

ekluzek · 2022-04-14T04:47:57Z

With the work in #1663 this work becomes obsolete so I'm closing. Some of the python code in here might still be useful examples that we keep in mind. One thing in here is one of our examples of interacting with the batch system.

negin513 added 9 commits July 16, 2021 03:28

testing commits

7a548ab

quick update

f27d072

ran the code through black...

3d73c19

making classes of remapper and mesh type

0106c33

more clean-ups

380e171

adding check_status + more features

9ac3e85

more cleanups...

046fa4a

cleaning up with black

80f708a

typo

2d8b2f6

negin513 requested a review from slevis-lmwg July 16, 2021 16:20

another typo

eb6da1f

slevis-lmwg reviewed Jul 16, 2021

View reviewed changes

ekluzek marked this pull request as draft September 2, 2021 19:36

slevis-lmwg mentioned this pull request Dec 24, 2021

Toolchain part1: ./gen_mksurf_namelist.py #1419

Merged

ekluzek added the closed: wontfix We won't fix this issue, because it would be too difficult and/or isn't important enough to fix label Apr 14, 2022

ekluzek closed this Apr 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Toolchain part2: `./mkmap_data.py` #1430

Toolchain part2: `./mkmap_data.py` #1430

negin513 commented Jul 16, 2021 •

edited by slevis-lmwg

Loading

slevis-lmwg left a comment

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

slevis-lmwg Jul 16, 2021

This comment was marked as off-topic.

slevis-lmwg commented Jul 23, 2021

ekluzek commented Feb 28, 2022

ekluzek commented Apr 14, 2022

	This Python script is part of the simplified toolchain for creating
	This Python script is STEP 2 of the simplified toolchain for creating


		3. Check if the weight (mapping file) already exists or not.

		4. If it does not exist, it creates the mapping (weight) file.

	To remove NPL from your environment on Cheyenne/Casper:
	To remove NPL (ncar_pylib) from your environment on Cheyenne/Casper:

	help="namelist for the ctsm case created by gen_mksurf_namelist.py ",
	help="namelist created by gen_mksurf_namelist.py ",

	# -- src mesh files
	# -- raw dataset files containing corresponding src mesh file names

Toolchain part2: ./mkmap_data.py #1430

Toolchain part2: ./mkmap_data.py #1430

Conversation

negin513 commented Jul 16, 2021 • edited by slevis-lmwg Loading

Description of changes

Specific notes

slevis-lmwg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as off-topic.

slevis-lmwg commented Jul 23, 2021

ekluzek commented Feb 28, 2022

ekluzek commented Apr 14, 2022

Toolchain part2: `./mkmap_data.py` #1430

Toolchain part2: `./mkmap_data.py` #1430

negin513 commented Jul 16, 2021 •

edited by slevis-lmwg

Loading