Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update NWM Client New interfaces and bring up to feature parity with old client #202

Closed
wants to merge 56 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
ca198f2
change from fastparquet to pyarrow
jarq6c Aug 2, 2022
6d7cebe
add unit handling module
jarq6c Aug 2, 2022
7c3843c
change import of unit handler test
jarq6c Aug 2, 2022
f8b48e8
make client interface more flexible
jarq6c Aug 2, 2022
0bc19a9
add reference time validation
jarq6c Aug 2, 2022
bdf9d3d
avoid redownloading files
jarq6c Aug 4, 2022
32cb2cd
reimplement parquet cache as parquet store
jarq6c Aug 4, 2022
cb1e767
tweak interface to remove canonicalization
jarq6c Aug 4, 2022
15a9e21
change defaults from parquet cache to store
jarq6c Aug 4, 2022
365120a
remove canonical mapping not generic enough
jarq6c Aug 4, 2022
d05bd2e
return all variables by default in processor
jarq6c Aug 4, 2022
090df56
add error checking
jarq6c Aug 4, 2022
37d4e91
add default variables to retrieve
jarq6c Aug 4, 2022
c6a2e79
use default features and variables in processor
jarq6c Aug 4, 2022
fe239ae
add append method
jarq6c Aug 4, 2022
ab8f48f
implement new get interface
jarq6c Aug 4, 2022
bdc810a
remove coordinates from default variables
jarq6c Aug 5, 2022
fc77637
correct logic error in data retrieval
jarq6c Aug 5, 2022
0100780
re-enable file clean-up
jarq6c Aug 5, 2022
57314b7
add unit handling
jarq6c Aug 5, 2022
bd905ee
add pint requirement
jarq6c Aug 5, 2022
b52205b
update parquet store tests
jarq6c Aug 5, 2022
8b49e4a
return coordinates with datasets from file processor
jarq6c Aug 5, 2022
0d28fba
update file processor tests
jarq6c Aug 5, 2022
6a54ad0
add file downloader test for overwrite
jarq6c Aug 5, 2022
2a0867e
validate reference time in get dataset
jarq6c Aug 5, 2022
1a716ce
avoid reassigning variables during processing
jarq6c Aug 5, 2022
d6b3585
update file client tests
jarq6c Aug 5, 2022
c6590d8
client no long relies on temp dirs
jarq6c Aug 5, 2022
6cc7c51
pass paths directly to file processor
jarq6c Aug 5, 2022
f908cde
mitigate xarray and or dask limitation with opening many files
jarq6c Aug 5, 2022
562b344
update file processor tests to match new interface
jarq6c Aug 8, 2022
4bea81e
use more descriptive naming scheme for files
jarq6c Aug 8, 2022
c8a67f4
test client.get_files
jarq6c Aug 8, 2022
3468ade
update docstrings
jarq6c Aug 8, 2022
473bda1
update examples in README
jarq6c Aug 8, 2022
cad7311
fix #209
aaraney Nov 18, 2022
ec90b60
bump nwis_client to 3.2.2
aaraney Nov 18, 2022
94e9c01
add test to verify 209 is fixed
aaraney Nov 18, 2022
789a8a9
add pytest-aiohttp dev dep
aaraney May 17, 2023
63e83cd
provide event loop to tests
aaraney May 17, 2023
f85e44d
pin importlib-metadata<=4.13.0. remove when 3.7 support dropped
aaraney May 19, 2023
30d78eb
Merge pull request #221 from aaraney/fix_211
jarq6c May 19, 2023
72dea01
remove infer_datetime parameter per pandas warning
jarq6c May 10, 2023
8a7331f
enable to option to return expanded metadata
jarq6c May 10, 2023
a85e67c
test expanded metadata option
jarq6c May 10, 2023
c8235e3
bump nwis client to 3.3.1
jarq6c May 10, 2023
64ea1d7
rename extra columns to match hydrotools canonical snake casing
jarq6c May 19, 2023
ff98895
check for new columns
jarq6c May 19, 2023
f21f04a
Merge pull request #218 from aaraney/fix-217
jarq6c May 19, 2023
b7489e1
Merge branch 'main' into fix-nwis-service-pandas-inplace-future-warni…
jarq6c May 19, 2023
6517e72
Merge pull request #210 from aaraney/fix-nwis-service-pandas-inplace-…
jarq6c May 19, 2023
0e8c85c
Echo dependency versions in runners (#220)
aaraney May 19, 2023
d542128
add note on installing on mac silicon. see #198 for full conversation…
aaraney Jun 8, 2023
a45d8a8
Merge branch 'nwm-new-units' of github.com:jarq6c/hydrotools into nwm…
jarq6c Jun 8, 2023
bbbcf17
remove 3.7 support
jarq6c Jun 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
pass paths directly to file processor
jarq6c committed Aug 5, 2022
commit 6cc7c51802314a61f837155b943e77232fbd9782
Original file line number Diff line number Diff line change
@@ -27,7 +27,7 @@ class NWMFileProcessor:
@classmethod
def get_dataset(
cls,
input_directory: Union[str, Path],
paths: Union[str, npt.ArrayLike],
feature_id_filter: npt.ArrayLike = _NWMClientDefault.CROSSWALK.index,
variables: List[str] = _NWMClientDefault.VARIABLES
) -> xr.Dataset:
@@ -50,20 +50,14 @@ def get_dataset(
-------
xarray.Dataset of input_directory lazily loaded.
"""
# Resolve input directory
input_directory = Path(input_directory)

# Generate file list
file_list = [f for f in input_directory.glob("*.nc")]

# Minimum coordinates
coordinates = []
for c in ["feature_id", "time", "reference_time"]:
if c not in variables:
coordinates.append(c)

# Open dataset
ds = xr.open_mfdataset(file_list, engine="netcdf4")
ds = xr.open_mfdataset(paths, engine="netcdf4")

# Prepare feature_id filter
if len(feature_id_filter) != 0: