Refactor significant portions of the codebase and refactor some land-atm handlers

[tomvothecoder]

Overview

This PR was originally supposed only address #115, but I found the codebase difficult to work with so I decided to refactor it.

Regression Testing

TLDR: All of the datasets produced by this branch align with the master branch. The max relative differences between 5/109 datasets are insignificant and can be attributed to floating point rounding error produced by Xarray vs. CDAT, which is fine.

---

Setup:

1. Generated branch datasets using scripts/branch-regression-testing/115-cdat-refactor-test/115-end-to-end-script.sh to compare against master branch datasets.
2. Compared datasets between branches using scripts/branch-regression/115-comparison-notebook.ipynb

Results:

• 104/109 are identical
• 4/109 are close and meet threshold of rtol=1e-7 (cl clw, cli, pfull)
• 1/109 is not close (mrso)
  • mrso is fine (max relative difference of 6.726486e-07)

    mrso
    -----
    MEAN
        dev: 998.3673095703125, master: 998.3673095703125
    SUM
        dev: 277705856.0, master: 277705856.0
    COMPARISON
    
    Not equal to tolerance rtol=1e-07, atol=1e-07
    
    Mismatched elements: 25556 / 777600 (3.29%)
    Max absolute difference: 0.00073242
    Max relative difference: 6.726486e-07
     x: array([[[nan, nan, nan, ..., nan, nan, nan],
            [nan, nan, nan, ..., nan, nan, nan],
            [nan, nan, nan, ..., nan, nan, nan],...
     y: array([[[nan, nan, nan, ..., nan, nan, nan],
            [nan, nan, nan, ..., nan, nan, nan],
            [nan, nan, nan, ..., nan, nan, nan],...

Summary of Changes

Core Changes

1. Refactor lib.py (now deleted)
   • Closes Refactor lib.py utilities to reduce code duplication and excessive nesting of function calls #190
   • Replace run_parallel() and run_serial() with __main__.py -> E3SMtoCMIP._run_parallel() and
     E3SMtoCMIP._run_serial()
     • Updated var name filepaths to vars_to_filepaths for clarity (its a dict mapping var key to list of string paths)
     • Update args passed to handler method, remove unnecessary args
     • Add try and except statement for submitting pool jobs to maintain compatibility with MPAS variable handlers, which use different handler method arguments
   • Replace handle_variables(), get_dimension_data() and load_axis() with VarHandler.cmorize()
   • Delete handle_simple() -- will be re-implemented from scratch in Fix --simple mode not working #130
2. Update handler.py
   • Update VarHandler.cmorize()
     • Refactored significantly -- extracted logic into smaller, maintainable private methods
     • Replaced data dictionary storing xr.DataArrays with ds (xr.Dataset object)
     • Distinguish CMOR usage with "cmor" string in function names and python variables
     • Add support for hybrid sigma variables: pfull, phalf
   • Add VarHandler._cmor_write()
     • Add support for CMORizing fixed-time variables
       • Separate PR for Refactor remaining land-atm fixed time variable handlers (areacella.py, orog.py, and sftlf.py) #217
     • If time dimension exists, CMORize the variable with all time and time bound values with a single call to cmor.write() instead of looping over each time value index and CMORizing each slice -- this should improve performance and removes the tqdm progress bar.
3. Update handlers.yaml
   • Add phalf and pfull entries
     • Closes Refactor phalf.py and pfull.py to handlers.yaml #115
     • Delete phalf.py and pfull.py
   • Add clcalispo entry
     • Closes Refactor clcalipso.py with handlers.yaml #218
     • Delete clcalipso.py
4. Update _formulas.py
   • Update all function argument types and return types from np.ndarray to xr.DataArray
   • Add formula functions for pfull and phalf
   • Add convert_units() function, which handles 1-to-1 unit conversions -- replaces default.default_handlers.write_data()
5. Refactor default.py (now deleted)
   • Replaced by VarHandler.cmorize() and _formulas.py.convert_units()

Clean Up Changes

7. Remove cdms2 and cdutil from dependencies in dev.yml and ci.yml7
8. Clean up legacy code in clisccp.py
   • Separate PR for Refactor clcalipso.py with handlers.yaml #218
9. Add Makefile for easy access to commonly used commands (e.g., building and installing package)

[tomvothecoder]

Moved the section below out of the description to make the description shorter.

Technical Debt in this Repo

I've noticed many Python anti-patterns and "not good" (bad) coding practices used in this repository. This has resulted in a lot of technical debt for new developers of this repo.

• A lot of copy-pasting code -- e.g., legacy Python modules for variable handlers, same logic in handle_variables()/handle_simple()/default_handler()
• Functions using a large number of arguments, especially keyword arguments with default values (example) (and no API docstrings or type annotations for any of these functions).
• Using **kwargs to pass around function arguments (example) -- really fragile and hard to debug
• Extremely large functions that should be broken up into smaller, maintainable functions
• No unit testing excepts the ones I implemented :(
• Functions and data structures are unnecessarily nested in one another (example, example)
• Excessive use of dictionaries to store handler information, variable and axes information, etc. (example)

[tomvothecoder]

Hi @chengzhuzhang, this PR is finally ready for review! I performed thorough regression testing and everything checks out.

The PR description includes the core changes that you might want to focus on. Thanks!

[tomvothecoder]

I added this TODO just in case it is needed. I didn't run into any issues where time bounds have a length of 1 and need to be reshaped like the old CDAT code linked in the comment.

[tomvothecoder]

Some last self-review

[chengzhuzhang]

Thanks for the note. I don't have a good idea of this either.

[chengzhuzhang]

@tomvothecoder The PR looks good to me. Thanks for bringing in these nice changes and carefully tested the changes. The code base is much cleaner now.

[tomvothecoder]

Thanks @chengzhuzhang! I will merge this now.