Enhance TC-Diag to use tc_diag_driver version 0.11.0

[JohnHalleyGotway]

Describe the Enhancement

On Dec 8, 2023, @robertdemariacira provided an updated version v0.11.0 of the tc_diag_driver (private repo). That update addresses the issues discussed in this comment for issue #2729.

This issue is basically a subtask of #2729, to be done immediately during the MET-12.0.0-beta3 development cycle. Further TC-Diag development will occur in the beta4 and beta5 cycles.

The tasks for this issue include:

• Update to tc_diag_driver v0.11.0.
  • Note that on 1/24/2024, we discovered that scipy really is a dependency for the tc_diag_driver code. It does run without it, but fails to compute several diagnostics (i.e. shear, tangential winds, and vorticity).
• [ Moved to beta4 ] Investigate why the computed diagnostics DO NOT MATCH @robertdemariacira. As discussed on 11/1/23:
  • It is OK if diag values differ by 1 due to rounding.
  • There are clearly problems with the winds.
  • @JohnHalleyGotway and @robertdemariacira will coordinate on testing/debugging.
  • See this comment for latest on debugging value differences.
• Investigate additional differences @jvigh notes in the PR review comment comparing MET and CIRA diag outputs:
  • Changes AVNO to GFSO. Yes, confirmed with @musgrave-kate on Nov 29, 2023 that this is desired.
  • Small but important differences in LAND and RMW values.
    • LAND diffs are likely OK because the definition of land has changed between when the CIRA diags were generated and now.
    • Need to investigate RMW differences through.
  • Spacing difference in the first column of the SOUNDING DATA output section. This is a problem and a space is needed.
  • Add checks to validate the length of the diagnostic names and unit strings. Error out if the names or units strings exceed 7 characters. Check with @musgrave-kate to confirm that this is the limit and not 8.
  • Units are missing (should be '/S') for 850VORT and 200DVRG. Confirm that once values are computed, units appear. @robertdemariacira will address this with an updated Python configuration option.
    • tc_diag_driver v0.11.0 does provide unit strings for the diagnostics even if the diagnostic values are bad data.
  • Should the number of MET vertical levels match those of CIRA? Review default configuration. Yes, confirmed this with @musgrave-kate on Nov 29, 2023.
  • The TGRD diagnostic is missing from the MET output.
    • After updating to tc_diag_driver v0.11.0 with scipy, it is now present.
• [ Moved to beta4 ] Update unit_tc_diag.xml to run a realistic HWRF parent + nest example. Be sure to test with with and without MET_PYTHON_EXE set. Use Hurricane Ian example for which we have diagnostics output and @KathrynNewman is providing gridded HWRF test data. Note that d01 should be used as the "parent" and d03 as the "nest" data source.
• Switch the default regridding method to be BILINEAR. That matches the existing Fortran and Python code functionality. However, test to confirm that the MET supports the regridding method being specified separately for each field being read.
• Provide configuration option to retain temporary NetCDF files for debugging purposes.
  • Incorporated MET_KEEP_TEMP_FILE environment variable to retain temp files.

Time Estimate

1 day.

Sub-Issues

Consider breaking the enhancement down into sub-issues.
None at this time.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2770043

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Review default alert labels
• Select component(s)
• Select priority
• Select requestor(s)

Milestone and Projects

• Select Milestone as the next official version or Backlog of Development Ideas
• For the next official version, select the MET-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
  No impacts currently.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issue
  Select: Milestone as the next official version
  Select: MET-X.Y.Z Development project for development toward the next official release
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

[JohnHalleyGotway]

After updating to tc_diag_driver version 0.11.0, I reran the unit_tc_diag.xml unit test but got a runtime error:

Traceback (most recent call last):
  File "/Volumes/d1/projects/MET/MET_development/MET-feature_2769_tc_diag_driver_v0.11.0/share/met/python/tc_diag/compute_tc_diag.py", line 218, in <module>
    main()
  File "/Volumes/d1/projects/MET/MET_development/MET-feature_2769_tc_diag_driver_v0.11.0/share/met/python/tc_diag/compute_tc_diag.py", line 96, in main
    config = post_resample_driver.config_from_file(
  File "/Users/johnhg/Library/Python/3.10/lib/python/site-packages/tc_diag_driver/post_resample_driver.py", line 62, in config_from_file
    spec = config_class(**filtered)
TypeError: DriverConfig.__init__() missing 1 required positional argument: 'atcf_tech_id'

@robertdemariacira I did a little digging but didn't see an obvious problem or solution. Can you please take a look and advise?

[robertdemariacira]

@JohnHalleyGotway That is an odd bug. It looks like you might be attempting to create a tc_diag_driver.model_files.ModelSpec instance using a config file intended to create a tc_diag_driver.post_resample_driver.DriverConfig object. Would you mind pointing me towards the version of compute_tc_diag.py you're working with here? Alternatively you can check that the call to config_from_file in compute_tc_diag.py looks like this:
config = post_resample_driver.config_from_file( config_filename, post_resample_driver.DriverConfig)

[JohnHalleyGotway]

@robertdemariacira sorry for the delay. The call to config_from_file looks correct. It hasn't changed. Here's a tar file with data/code to demonstrate the runtime issue:
for_robert.tar.gz

musial6:for_robert johnhg$ python3 compute_tc_diag.py post_resample.yml v2023-04-07_gdland_table.dat tmp_tc_diag_AL092022_GFSO_2022092400_f0_parent_13052_0_.nc
Traceback (most recent call last):
  File "/Volumes/d1/projects/MET/MET_development/MET-feature_2769_tc_diag_driver_v0.11.0/internal/test_unit/for_robert/compute_tc_diag.py", line 218, in <module>
    main()
  File "/Volumes/d1/projects/MET/MET_development/MET-feature_2769_tc_diag_driver_v0.11.0/internal/test_unit/for_robert/compute_tc_diag.py", line 96, in main
    config = post_resample_driver.config_from_file(
  File "/Users/johnhg/Library/Python/3.10/lib/python/site-packages/tc_diag_driver/post_resample_driver.py", line 62, in config_from_file
    spec = config_class(**filtered)
TypeError: DriverConfig.__init__() missing 1 required positional argument: 'atcf_tech_id'