Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update NWM Client New interfaces and bring up to feature parity with old client [Updated] #227

Merged
merged 26 commits into from
Jun 13, 2023

Conversation

jarq6c
Copy link
Collaborator

@jarq6c jarq6c commented Jun 13, 2023

This turned out to be a bit bigger than I had intended, but it is the result of months of experimentation as well as feedback from frequent users of National Water Model output. I'll summarize changes for each component.

FileDownloader.py

Added an overwrite boolean parameter that will warn and skip similarly named files.

NWMFileProcessor.py

Updated get_dataset to accept the xarray.open_mfdataset paths argument. Essential coordinates (reference_time, time, and feature_id) are always returned.

NWMClient.py

Updated abstract interface to accept an nwm_feature_ids parameter to facilitate retrieval of smaller amounts of data. Unfortunately, retrieving data is still relatively slow, but this new interface accommodates the most common use-case (retrieval of time series data from a few sites).

NWMFileClient.py

Added unit_system parameter to match older nwm_client unit handling. Replaced get_dataset method with get_files that just downloads files and returns their local paths. get method works with several configurations and reference_times by default. Implements nwm_feature_ids parameter.

ParquetStore.py

Formerly ParquetCache.py, I decided to rename this module and reimplement as a MutableMapping similar to pandas.HDFStore. It can still be used as a context manager, but now it also includes key-access and inherits useful methods like get and pop from MutableMapping. As a consequence, the old get method was removed. This could live somewhere else, eventually.

NWMClientDefaults.py

Added defaults for handling units, variables, and the new ParquetStore.

UnitHandler.py

Taken verbatim from the older nwm_client. This could potentially live somewhere else (eventually).

_version.py

bumped to 7.0.0 due to breaking interface changes

Checklist

  • PR has an informative and human-readable title
  • PR is well outlined and documented. See #12 for an example
  • [?] Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (see CONTRIBUTING.md)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output) using numpy docstring formatting
  • Placeholder code is flagged / future todos are captured in comments
  • Reviewers requested with the Reviewers tool ➡️

@jarq6c jarq6c changed the title Nwm new units fix Update NWM Client New interfaces and bring up to feature parity with old client [Updated] Jun 13, 2023
@jarq6c jarq6c requested a review from aaraney June 13, 2023 13:52
@jarq6c jarq6c self-assigned this Jun 13, 2023
@jarq6c jarq6c added bug Something isn't working enhancement New feature or request labels Jun 13, 2023
@jarq6c jarq6c merged commit 8844904 into NOAA-OWP:main Jun 13, 2023
@jarq6c jarq6c deleted the nwm-new-units-fix branch June 14, 2023 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant